TL;DR
This paper introduces "Ignition," a hybrid on-demand recruiting system that balances immediate worker recruitment with a retainer pool to achieve low-latency crowdsourcing over long deployments. Their 10-month study demonstrates that this approach effectively balances cost and speed, providing a stable and responsive workforce for interactive crowd-powered applications.
What Problem Does This Paper Address?
Interactive crowd-powered systems, which rely on human input for tasks that automated solutions can't handle, face a critical challenge: latency. This latency can stem from getting workers to the task, workers completing the task, or integrating their work. While the latter two are task-specific, recruiting workers quickly is a universal hurdle. Existing solutions include posting tasks on-demand (cheaper but slower) or maintaining a retainer pool (faster but more expensive). However, most studies evaluate these methods over short periods, failing to account for how worker availability and marketplace dynamics change over time. This paper addresses the need for a robust, cost-effective, and stable on-demand recruiting strategy for long-term deployments of low-latency crowdsourcing systems.
Methods
The core contribution is the "Ignition" framework, which combines two recruitment strategies:
- On-Demand Recruiting (Marketplace): Posting Human Intelligence Tasks (HITs) on platforms like Amazon Mechanical Turk (MTurk) when a task needs workers.
- Retainer Pool: Recruiting workers in advance and having them wait in a dedicated page, ready to be called upon quickly.
Ignition's key innovation lies in its hybrid approach: when a task arrives, it first attempts to recruit a small number of workers from the retainer pool. If more workers are needed, it then posts HITs on MTurk. Crucially, Ignition aims to recruit slightly more workers than immediately required. Those who arrive after the task is "fully-occupied" are directed to a waiting page, becoming part of the retainer pool for future tasks. This dynamic allocation aims to balance the speed of the retainer with the cost-effectiveness of the marketplace. The system also includes a one-minute interactive tutorial for new workers and a penalty for those who don't respond to alerts within 20 seconds.
Datasets
The study reports on a 10-month deployment of Ignition supporting the "Chorus" conversational agent. The data collected includes:
- 745 conversational sessions (tasks)
- 6,823 assignments from 648 workers
- Worker response times to recruitment alerts
- Worker recall rates from the retainer pool
- Worker survey responses from 101 participants who completed at least 10 HITs.
No new benchmark datasets were introduced; the study leverages real-world usage data from a deployed system.
Key Insights and Ideas
- Hybrid is Key: Combining retainer and marketplace recruitment is crucial for balancing speed and cost over long deployments. Relying solely on one method can be either too slow or too expensive.
- Over-recruitment Pays Off: Intentionally recruiting more workers than immediately needed for a task effectively populates the retainer pool for subsequent tasks, ensuring faster future recruitment.
- Marketplace Dynamics Matter: The number of assignments posted for a HIT significantly impacts recruitment speed. Larger HITs are more likely to attract workers faster, confirming the assumption that more marketplace effort leads to quicker acquisition.
- Retainer Stability: The retainer model proved to be reliably effective, recalling 80-90% of workers within seconds, even over a long deployment period.
- Worker Preferences: While workers generally prefer regular HITs, they appreciate the retainer model for its clear timers, bonus potential, and the ability to work on other tasks in parallel. However, they dislike the lower payment for waiting time and the strict response deadlines.
- Cost-Benefit Trade-off: The paper highlights the significant cost of maintaining a retainer pool, emphasizing that Ignition's dynamic approach, which avoids a fixed retainer cost, is more economically viable for long-term deployments.
Key Results
- Stable Retainer Recall: The response rate for workers recalled from the retainer was consistently high (above 80%) for most of the 10-month deployment, demonstrating its reliability.
- Faster Marketplace Recruitment with More Assignments: Posting a HIT with 8 assignments, compared to 5, increased the probability of getting at least one worker within 80 seconds from 40% to 79%.
- Worker Engagement: 34.7% of surveyed workers liked getting paid for waiting, and 29.7% appreciated the clear interface design of the waiting page.
- Worker Dissatisfaction: The primary complaint from workers was the low payment for waiting time (average dislikability rating of 3.64/5), followed by the strict 20-second response requirement.
- Cost Efficiency: Ignition's dynamic strategy, which adjusts marketplace recruitment based on retainer availability, helps manage costs compared to a fixed-size retainer.
Limitations and Future Work
- Task Completion Not Measured: The study primarily focused on recruitment latency, not on worker performance or task completion rates. It's acknowledged that even with fast recruitment, workers might still return HITs or not complete them.
- Marketplace Load: The study was conducted on a medium-sized deployment. Future work should explore how findings change under increased marketplace load or with larger, more active deployments.
- Worker Behavior Adaptation: The paper suggests investigating how worker response times and recruitment might evolve over time as workers become more familiar with the tasks and the market changes.
- Continuity and Micro-diversions: Incorporating strategies like micro-diversions to keep workers engaged longer could be explored.
- Platform Specificity: The findings are based on MTurk; future research could examine how these strategies perform on different platforms.
- Advanced Modeling: Developing models to learn optimal recruitment policies based on budget, worker behavior, and task characteristics is a promising direction.
Summary updated