Robot Scavenger Hunt: A Standardized Framework for Evaluating Intelligent Mobile Robots

Shiqi Zhang, Dongcai Lu, Xiaoping Chen, Peter Stone

Abstract

The Robot Scavenger Hunt is a benchmarking framework designed to evaluate intelligent mobile robots in real-world, long-term scenarios. Unlike traditional short-duration competitions in controlled environments, this framework challenges robots to complete diverse tasks across multi-floor buildings over several hours. It emphasizes autonomous scheduling, robust environmental adaptation, and decentralized evaluation through a cloud-based certificate system.

1. Introduction

While robotics has advanced significantly in AI capabilities and hardware, standardized comparison remains a challenge. Most existing benchmarks, such as RoboCup@Home, operate in highly controlled, small-scale environments over short durations. These "sprint-style" evaluations fail to capture the complexities of true service robots, which must operate in "marathon" conditions—navigating crowded hallways, interacting with non-expert humans, and managing time effectively over extended periods. The Robot Scavenger Hunt addresses this by providing a standardized, decentralized framework for testing robots in their native environments while enabling global comparison.

2. Methodology

The framework is built upon three core components:

2.1 The Task Library

Standardized challenges are defined as four-tuples: <name, certificate format, score, description>. Completion is verified via "completion certificates" (e.g., trajectory maps or timestamped photos) uploaded to a cloud service, ensuring objective evaluation.

2.2 The Object Library

To maintain consistent difficulty across different vision systems, the framework utilizes a list of globally accessible common objects (e.g., tea cups, soda cans). This prevents bias toward specific detection algorithms.

2.3 Autonomous Scheduling

A key innovation is the requirement for robots to autonomously prioritize tasks to maximize their score. This forces robots to reason about environmental context, such as human traffic patterns. To ensure variety, a minimum time gap (currently one hour) is enforced between repetitions of the same task.

3. Evaluation Framework

The Robot Scavenger Hunt distinguishes itself from benchmarks like RoboCup@Home by focusing on larger spaces (multi-floor buildings), longer durations (hours), and real-world research environments with actual residents. The decentralized nature allows teams to participate from their home labs, lowering the barrier to entry for international competition.

4. Experimental Results

The framework's viability was demonstrated using two distinct robot platforms: the KeJia Robot from USTC and the BWIBots from UT Austin.

The KeJia (left) and BWIBot (right) platforms used to validate the Scavenger Hunt framework.

The KeJia robot utilizes a 5-DOF arm and a natural language dialogue module for task planning. In contrast, the BWIBots are built on Segway bases and excel at multi-floor navigation using symbolic planning. Despite their hardware and software differences, both platforms successfully participated under the same rules and verification system.

5. Conclusion and Future Work

The current framework serves as a foundational step toward long-term robot evaluation. Future work includes expanding the task and object libraries, refining scheduling constraints, and introducing more complex human-robot interaction (HRI) tasks that require deeper semantic understanding of human intent.

Summary updated

Abstract

In recent years, many different types of intelligent mobile robots have been developed in research and industrial labs. Although there are significant differences in both hardware and software over these robots, many of them share a common set of AI capabilities, e.g., planning, learning, vision and natural language processing. At the same time, almost all of them are equipped with traditional robotic capabilities such as mapping, localization, and navigation. However, to date it has been difficult to compare and contrast their capabilities in any controlled way. The main goal of the Robot Scavenger Hunt is to provide a standardized framework that includes a set of standardized tasks for evaluating the AI and robotic capabilities of medium-sized intelligent mobile robots. Compared to existing benchmarks, e.g., RoboCup@Home, Robot Scavenger Hunt aims at evaluations in larger spaces (multi-floor buildings vs. rooms) over longer periods of time (hours vs.minutes) while interacting with real human residents.
BibTeX
@inproceedings{zhang2016robot,
  author = {Shiqi Zhang and Dongcai Lu and Xiaoping Chen and Peter Stone},
  title = {{Robot Scavenger Hunt: A Standardized Framework for Evaluating Intelligent Mobile Robots}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure