Introduction
Robots operating in dynamic, real-world environments frequently encounter objects and scenarios that fall outside their pre-trained recognition capabilities. While local machine learning continues to advance, it often lacks the broad semantic vocabulary and "common sense" inherent to human observers. NIMBUS (Networked Infrastructure for Multi-user Behavioral Urban Systems) addresses this gap by providing a hybrid architecture that allows robots to outsource visual labeling tasks to a distributed human workforce in near real-time.
The primary challenge in such human-in-the-loop systems is balancing the trade-off between response latency and data quality. For a robot engaged in social interaction, timely feedback is critical to maintaining the flow of the engagement. However, crowdsourced workers—such as those on Amazon Mechanical Turk (AMT)—exhibit variable reliability and response times.
The NIMBUS Architecture
The NIMBUS system acts as a mediator between a robot's "Front-End" and a "Back-End" composed of cloud and crowd resources. The Front-End handles the robot's immediate perception-action loop, utilizing deictic gestures (pointing) and gaze to identify areas of interest. When the robot encounters an unknown stimulus, it captures a scene, crops relevant image patches, and uploads them to the NIMBUS server farm.
NIMBUS manages the "Mechanical Turk Wrapper," which formats the robot's request into a Human Intelligence Task (HIT). This wrapper provides workers with specific instructions, such as "Can you find the object in my hand?", and collects the resulting bounding box labels.
Labeling Strategies: Speed vs. Quality
To investigate the latency-quality trade-off, the authors implemented two distinct labeling strategies:
- One-Shot Labeling: This strategy prioritizes speed by sending an image to a single AMT worker and accepting the first response received.
- Rollover Labeling: This strategy prioritizes data fidelity. The system takes the initial bounding box from the first worker and "rolls it over" to a second worker for verification or refinement. This iterative process acts as a filter to reduce noise and tighten the bounding box around the target object.
Experimental Evaluation and Results
The system was evaluated through in-situ trials involving a live robot and real AMT workers. The robot used ImageNet as a backend resource for initial figure-ground patch discovery to isolate objects before soliciting human labels. Accuracy was measured by calculating the Mean Squared Error (MSE) of the crowdsourced bounding boxes against a ground-truth baseline.
The results highlighted a stark contrast between the two strategies. One-Shot labeling achieved a median response time of 37.5 seconds, with some responses returning in as little as 13 seconds. However, this speed came at the cost of higher error (MSE of 518.6px) and greater variance.
In contrast, Rollover labeling significantly improved data quality, reducing the MSE to 341.5px and lowering the standard deviation. This increased precision is vital for the robot to "reify" a visual stimulus into a stable, long-term object detector. However, the median response time ballooned to approximately 4.4 minutes (263.58 seconds), which is generally too slow for fluid social interaction.
Conclusion and Future Work
NIMBUS demonstrates the feasibility of integrating crowdsourced intelligence into robotic cognitive architectures. While the "13-second dream" of One-Shot labeling approaches the realm of interactive learning, the 4-minute latency of the more reliable Rollover strategy remains a significant bottleneck.
Future research will focus on mitigating this "social latency" through worker modeling—identifying and routing tasks to highly reliable individuals—and exploring parallel processing strategies. By asking multiple workers simultaneously rather than sequentially, robots may be able to statistically eliminate outliers and achieve high-quality labels without the prohibitive delays of sequential verification.
Summary updated