Human-centered In-building Embodied Delivery Benchmark

Zhuoqun Xu, Yang Liu, Xiaoqi Li, Jiyao Zhang, Hao DongarXiv 2024 · DOI · arXiv

Introduction

Current research in Embodied AI (EAI) often focuses on isolated, skill-driven tasks within static environments. However, a significant "reality gap" persists between these laboratory settings and the requirements of real-world commercial delivery. In complex indoor environments like offices or hospitals, robots must navigate multi-floor architectures, interact with dynamic human targets, and interpret nuanced natural language instructions. To address these challenges, researchers have introduced the Human-centered In-building Embodied Delivery Benchmark, a comprehensive simulation environment and dataset designed for end-to-end delivery tasks.

The Polar Research Station (PRS) Environment

The benchmark is built upon the Polar Research Station (PRS) Simulator, developed using Unity and the PhysX engine. This three-story environment includes diverse functional spaces such as laboratories, gyms, and offices, connected by a functional elevator system. Unlike many grid-based simulators, PRS utilizes continuous physics for more realistic interactions.

Overview of the PRS environment, including the layout of the three-story facility, diverse object categories, and autonomous NPC characters.

A defining feature of the PRS environment is the presence of ten autonomous Non-Player Characters (NPCs). These characters follow distinct daily routines driven by Large Language Models (LLMs), moving independently throughout the building. This creates a "moving target" problem, requiring the robot to not only locate an object but also actively search for the requester who may have changed locations since the initial instruction.

Methodology and Task Pipeline

The delivery task is framed as a multi-stage sequence: instruction parsing, object search, manipulation, navigation, and human delivery. The proposed baseline system utilizes a modular architecture to handle these stages.

The end-to-end delivery pipeline: the robot searches for the requested object, performs a grasp, navigates through the building, searches for the target NPC, and completes the delivery.

Key technical components include:

  1. Instruction Parsing: An LLM (e.g., GPT-4) decomposes complex natural language into executable sub-tasks.
  2. Visual Scene Understanding: A Large Multimodal Model (LMM) provides common-sense reasoning, helping the robot identify likely "receptacles" (e.g., finding a bottle on a dining table) when an object is not immediately visible.
  3. Navigation and Manipulation: The system employs an A* algorithm for path planning and a 6-DOF grasping API for physical interaction within the continuous physics space.

Dataset and Evaluation

The PRS dataset comprises 13,296 language instructions, meticulously refined to ensure grammatical correctness and diversity. Unlike existing benchmarks such as ALFRED or Habitat, PRS is the first to integrate multi-floor navigation, autonomous NPCs, and human-centered task objectives simultaneously. Evaluation focuses on "zero-shot" reasoning, where models must generalize to the environment without specific prior training, mimicking the deployment conditions of commercial robots.

Experimental Results

Baseline tests using a combination of GPT-4o and Grounding DINO reveal the complexity of the benchmark. While the models excel at instruction parsing (71.7% success rate) and finding the target human (88.7%), the overall task success rate (SR) remains low at 32.2%.

Metric Success Rate (%)
Overall Task Success (SR) 32.2
Instruction Parsing 71.7
Manipulation (Grasping) 59.2
Human Search 88.7

The primary bottlenecks are the initial object search and the physical challenges of manipulation, highlighting the difficulty of chaining long sequences of perception, reasoning, and execution in dynamic settings.

Conclusion and Future Directions

The Human-centered In-building Embodied Delivery Benchmark provides a rigorous testing ground for the next generation of delivery robots. By modeling the complexities of real-world buildings and human behavior, it exposes the limitations of current state-of-the-art models. Future work will focus on expanding architectural diversity, increasing NPC social complexity, and incorporating multi-modal human-robot interaction to further bridge the reality gap.

Summary updated

BibTeX
@misc{xu2024human,
  author = {Zhuoqun Xu and Yang Liu and Xiaoqi Li and Jiyao Zhang and Hao Dong},
  title = {{Human-centered In-building Embodied Delivery Benchmark}},
  year = {2024},
  eprint = {2406.17898},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.17898},
}
Figure