Abstract
Human computation allows computer systems to leverage human intelligence in computational processes. While it has primarily been used for tasks that are not time-sensitive, recent systems use crowdsourcing to get on-demand, real-time, and even interactive results. In this paper, we present techniques for building real-time crowdsourcing systems, and then discuss how and when to use them. Our goal is to provide system builders with the tools and insights they need to replicate the success of modern systems in order to further explore this new space.
Introduction: Overcoming the Latency Barrier
Traditionally, crowdsourcing platforms like Amazon Mechanical Turk have operated as asynchronous, "offline" processes. Requesters post tasks and wait hours or even days for workers to complete them. While effective for large-scale data labeling, this model fails for interactive applications requiring immediate feedback.
Consider a blind user needing to identify a grocery item or a robot requiring human intervention to navigate a sudden obstacle. In these scenarios, the "latency barrier"—the time it takes to recruit and synchronize human workers—is the primary obstacle. To address this, we propose an architectural framework for "crowd-powered" systems that can respond in seconds, effectively transforming a distributed online crowd into a reliable, real-time agent.
A Unified Architecture for Real-Time Crowdsourcing
Building a real-time crowd-powered system requires a coordinated pipeline that manages the flow from an initial request to a final, aggregated answer. The architecture must handle task decomposition, worker recruitment, and the synthesis of noisy inputs into a coherent output.
As shown in the architecture above, the process begins with Task Division. Complex problems are split using Stream Parallelism (multiple workers performing the same task for redundancy) or Temporal Division (workers taking short "shifts," essential for continuous tasks like live captioning).
To ensure sub-second response times, systems often employ Synchronous Recruitment via "retainer" models. Workers are paid a small fee to remain on-call, ensuring they are ready the instant a task arrives. Once active, workers (W1 through Wn) are guided by a Reward Scheme and provided with Context Maintenance. Because workers join and leave dynamically, the system must provide explicit instructions or allow them to see the results of previous actions (implicit context) so they can contribute effectively without starting from scratch.
Navigating the Speed-Reliability Trade-off
A central challenge in architecting these systems is balancing the competing demands of speed, reliability, and cost. Different workforce models occupy different niches within this space.
As illustrated in the trade-off matrix, Open Crowds (such as social media volunteers) offer high speed but often lower reliability. Conversely, Expert Volunteers and Paid Workforces (like call centers) provide higher reliability but may be slower to mobilize or more expensive. The ultimate goal for many systems is to move toward the "Well-trained automatic system" quadrant, where AI can eventually replicate human performance at scale.
Key Insights and System Performance
Our analysis of landmark systems like VizWiz (visual questions for the blind), Scribe (real-time captioning), and Legion (continuous robot control) reveals several critical insights:
- Response Aggregation: To mitigate noise, systems must decide when to accept an answer. Waiting for consensus increases reliability but adds latency. Optimizing this "frontier" is key to system performance.
- Implicit Communication: In continuous control tasks, workers perform better when they see the outcome of the collective action (e.g., a robot moving) rather than the raw inputs of other workers, which can lead to circular errors.
- AI as a Scaffold: Crowdsourcing is not merely a stopgap for immature AI; it is a training mechanism. Real-time systems provide live training data, allowing automated components to gradually take over more of the task.
Empirical results prove this architecture's viability. Scribe demonstrated that a group of non-experts can achieve near-professional captioning accuracy with less than five seconds of latency. Similarly, Legion proved that a crowd could collectively control a complex user interface for over an hour, despite high worker turnover.
Future Directions
While current architectures are robust, challenges remain in reducing the cost of maintaining "warm" crowds and improving long-term context retention across different worker groups. Future work should focus on developing "Expert Crowds" for specialized domains and creating seamless, invisible hand-offs between human agents and automated systems, moving closer to true hybrid intelligence.
Summary updated