Abstract
The Robot Scavenger Hunt is a benchmarking framework designed to evaluate intelligent mobile robots in real-world, long-term scenarios. Unlike traditional short-duration competitions in controlled environments, this framework challenges robots to complete diverse tasks across multi-floor buildings over several hours. It emphasizes autonomous scheduling, robust environmental adaptation, and decentralized evaluation through a cloud-based certificate system.
1. Introduction
While robotics has advanced significantly in AI capabilities and hardware, standardized comparison remains a challenge. Most existing benchmarks, such as RoboCup@Home, operate in highly controlled, small-scale environments over short durations. These "sprint-style" evaluations fail to capture the complexities of true service robots, which must operate in "marathon" conditions—navigating crowded hallways, interacting with non-expert humans, and managing time effectively over extended periods. The Robot Scavenger Hunt addresses this by providing a standardized, decentralized framework for testing robots in their native environments while enabling global comparison.
2. Methodology
The framework is built upon three core components:
2.1 The Task Library
Standardized challenges are defined as four-tuples: <name, certificate format, score, description>. Completion is verified via "completion certificates" (e.g., trajectory maps or timestamped photos) uploaded to a cloud service, ensuring objective evaluation.
2.2 The Object Library
To maintain consistent difficulty across different vision systems, the framework utilizes a list of globally accessible common objects (e.g., tea cups, soda cans). This prevents bias toward specific detection algorithms.
2.3 Autonomous Scheduling
A key innovation is the requirement for robots to autonomously prioritize tasks to maximize their score. This forces robots to reason about environmental context, such as human traffic patterns. To ensure variety, a minimum time gap (currently one hour) is enforced between repetitions of the same task.
3. Evaluation Framework
The Robot Scavenger Hunt distinguishes itself from benchmarks like RoboCup@Home by focusing on larger spaces (multi-floor buildings), longer durations (hours), and real-world research environments with actual residents. The decentralized nature allows teams to participate from their home labs, lowering the barrier to entry for international competition.
4. Experimental Results
The framework's viability was demonstrated using two distinct robot platforms: the KeJia Robot from USTC and the BWIBots from UT Austin.
The KeJia robot utilizes a 5-DOF arm and a natural language dialogue module for task planning. In contrast, the BWIBots are built on Segway bases and excel at multi-floor navigation using symbolic planning. Despite their hardware and software differences, both platforms successfully participated under the same rules and verification system.
5. Conclusion and Future Work
The current framework serves as a foundational step toward long-term robot evaluation. Future work includes expanding the task and object libraries, refining scheduling constraints, and introducing more complex human-robot interaction (HRI) tasks that require deeper semantic understanding of human intent.
Summary updated