RoboCup@Home - Benchmarking Domestic Service Robots

Sven Wachsmuth, Dirk Holz, Maja Rudinac, Javier Ruiz-del-SolarAAAI 2015 · DOI

TL;DR

RoboCup@Home is a "living benchmark" designed to evolve alongside the state-of-the-art in domestic service robotics. By analyzing yearly performance data across core capabilities like navigation and speech recognition, the competition organizers dynamically adjust rules to ensure the challenge remains difficult enough to drive genuine research breakthroughs in real-world human-robot interaction.

The Challenge of Domestic Benchmarking

Evaluating domestic service robots is notoriously difficult. Unlike specialized industrial robots, home assistants must integrate a massive array of skills—mapping, person tracking, object manipulation, and natural language processing—into a single, fluid behavior.

Traditional laboratory evaluations often fail to capture the "messiness" of real-world environments. Furthermore, static benchmarks in robotics tend to suffer from stagnation: once a specific task is "solved," the benchmark no longer incentivizes further innovation. RoboCup@Home addresses the need for a benchmarking process that is both comprehensive and adaptive, capable of guiding the robotics community from simple lab tasks toward complex, autonomous operation in unpredictable human environments.

Methodology: A Multi-Stage Framework

The competition serves as a multi-stage benchmarking framework with several key components:

  • Two-Stage Competition: Robots first prove "Basic Functionalities" (Stage 1) such as following a person or reacting to an emergency. Successful teams move to Stage 2, which involves complex, integrated tasks like hosting a "Cocktail Party" (finding people and delivering specific drinks) or acting as a "General Purpose Service Robot" (handling inconsistent or partially defined verbal commands).
  • Binary Scoring of Sub-tasks: Instead of a single "pass/fail" grade for a task, each task is broken down into sub-tasks scored binarily. This allows the Technical Committee (TC) to identify exactly which capability (e.g., gesture recognition vs. path planning) is the current bottleneck for the field.
  • Dynamic Rule Evolution: The core method is a continuous feedback loop. The TC analyzes the performance of the best teams each year. If a specific capability reaches a high success rate across the league, the rules for the following year are made more stringent to push the boundaries further.

Datasets and Scenarios

While not a traditional machine learning paper based on a static digital dataset, this work introduces a physical benchmarking "dataset" in the form of the yearly RoboCup@Home Rulebook. The "data" consists of standardized domestic environments (living rooms, kitchens) and public spaces (restaurants) where robots are tested. The paper utilizes a longitudinal dataset of competition results from 2008 to 2014, tracking the success rates of the "best-in-class" teams across nine specific robotic capabilities.

Key Insights and Ideas

  • The "Moving Goalposts" Strategy: A benchmark should get harder over time. For example, in 2009, mapping and speech recognition were no longer "points of failure" for top teams. In response, organizers introduced "supermarket" tests in unknown environments and banned head-mounted microphones, forcing robots to use on-board arrays to handle ambient noise.
  • Integration as the Primary Hurdle: While individual components (like vision algorithms) might work in a lab, the primary "show stopper" is usually the coordination of these skills. The competition forces researchers to move beyond "siloed" AI.
  • Transitioning to Open Worlds: The league has successfully transitioned from "closed" environments (where the robot knows the floor plan in advance) to "open" environments (where the robot must navigate a previously unknown restaurant or public space).

Key Results

Analysis of performance trends from 2008–2014 reveals the evolving landscape of the field:

Performance trends of robotic capabilities from 2008 to 2014 alongside a robot performing a manipulation task.
  • Maturity of Navigation: Navigation and mapping reached near 100% performance by 2009, leading to the introduction of more complex obstacle avoidance and "unknown" maps.
  • The Difficulty of Manipulation: Object manipulation remains one of the most volatile and difficult skills, with performance fluctuating significantly as the variety and placement of objects become more realistic.
  • The 2013-2014 "Dip": There is a visible drop in performance across several categories in 2013 and 2014. This was not due to declining robot quality, but rather the Technical Committee significantly raising the bar for autonomy and reducing the amount of "pre-knowledge" teams could provide their robots.

Limitations and Future Work

Several hurdles remain for domestic robotics:

  • The "Out of the Box" Requirement: Currently, robots still require significant setup time in a new environment. Future iterations aim to reward robots that can function with zero prior setup.
  • Robustness to Failure: Most current systems are "brittle." Future benchmarks will incentivize behavior monitoring and more sophisticated error recovery.
  • Long-term Autonomy: The competition currently focuses on short bursts of activity. The next step is requiring robots to operate for much longer periods without human intervention or resource management issues.

Summary updated

BibTeX
@inproceedings{wachsmuth2015robocup,
  author = {Sven Wachsmuth and Dirk Holz and Maja Rudinac and Javier Ruiz-del-Solar},
  title = {{RoboCup@Home - Benchmarking Domestic Service Robots}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2015},
  doi = {10.1609/aaai.v29i1.9350},
}
Figure