Introduction
Traditional Artificial Intelligence (AI) research has historically excelled in "toy domains"—narrow, well-defined environments where variables are controlled and edge cases are limited. However, transitioning these models into the messy, unpredictable real world remains a significant hurdle. Many systems fail to reach deployment because they cannot reliably handle the "long tail" of rare or complex scenarios.
This research proposes a "scaffolding" approach to AI development. Rather than waiting for automation to reach perfection, we can utilize real-time human computation to power the components that current algorithms cannot yet manage. By treating a crowd of non-experts as a single, reliable "crowd agent," we can deploy functional systems today that simultaneously generate the high-quality, in situ data required to train the fully automated models of tomorrow.
The Continuous Real-Time Crowdsourcing Model
The foundation of this approach is continuous real-time crowdsourcing. Unlike asynchronous platforms like Amazon Mechanical Turk, where tasks are discrete and static, this model involves a synchronous collective of workers contributing to a live, evolving task. To make this crowd behave like a single, coherent agent, the system employs several technical mediators:
- The Input Mediator: For control-oriented tasks, such as robot navigation, the system must resolve conflicting inputs from multiple users. It utilizes cosine similarity to weight workers based on their alignment with the group consensus. By selecting a "leader" for sub-second intervals, the system produces a smooth, reliable control stream.
- Multiple Sequence Alignment: In transcription tasks (e.g., Legion:Scribe), the system uses an $A^*$ search algorithm to merge partial, overlapping captions from multiple non-experts into a single, high-fidelity text stream in real-time.
- Hybrid Intelligence: Systems like Legion:AR combine automated models (such as Hidden Markov Models) with human oversight. When the automated model’s confidence drops below a certain threshold, it triggers a request for crowd labels. These labels are then used to retrain the model on the fly, allowing the AI to learn from the specific environment in which it is deployed.
Key Systems and Applications
The efficacy of the crowd-agent abstraction has been demonstrated across several domains, showing that collective human intelligence can match or exceed specialized expertise.
Real-Time Transcription (Legion:Scribe)
Legion:Scribe enables a group of 3–5 non-experts to provide live captions for natural speech. While professional stenographers require years of training to reach speeds of 250+ words per minute (wpm), the Legion:Scribe system allows a small crowd to achieve the same result by merging their collective inputs. This exploits an asymmetry in human-machine strengths: while humans handle context and common words well, the system can eventually integrate Automated Speech Recognition (ASR) to handle phonetically unique terms.
Conversational Assistance (Chorus)
The Chorus system provides a conversational interface powered by the crowd. In testing, Chorus:Assist correctly answered over 84% of user prompts, including complex follow-up queries that typically baffle standard digital assistants. A variant, Chorus:View, was developed to assist blind users. By maintaining a persistent "crowd agent," the system reduced the latency for visual question-answering from several minutes to just a few seconds.
Insights: The Crowd as a Scaffold
A central insight of this work is the concept of organizational memory. The crowd, as a collective, can retain information that individual members might forget, functioning similarly to a small society. This collective memory allows the system to maintain context over long interactions.
Furthermore, this approach facilitates Training in the Wild. By deploying systems that use the crowd as a scaffold, researchers can collect training data from actual use cases rather than laboratory simulations. This ensures the underlying AI is exposed to the exact edge cases and environmental nuances it will face once the human scaffold is eventually removed.
Future Directions
While the scaffolding approach allows for immediate deployment, the ultimate goal is seamless automation. Future research is focused on:
- Formalizing Action Logic: Developing tools like "ARchitect" to extract STRIPS-style pre- and post-conditions from crowd labels, helping AI understand the underlying logic of tasks.
- Seamless Blending: Refining the hand-off between human and machine to ensure users do not perceive a shift in performance or latency.
- Generalization: Creating a universal framework where any AI system can actively learn from a real-time crowd, regardless of the specific domain.
By bridging the gap between human intelligence and machine automation, real-time crowdsourcing provides a viable path toward truly deployable, intelligent systems that evolve alongside their users.
Summary updated