Introduction
Current research in Embodied AI (EAI) often focuses on isolated, skill-driven tasks within static environments. However, a significant "reality gap" persists between these laboratory settings and the requirements of real-world commercial delivery. In complex indoor environments like offices or hospitals, robots must navigate multi-floor architectures, interact with dynamic human targets, and interpret nuanced natural language instructions. To address these challenges, researchers have introduced the Human-centered In-building Embodied Delivery Benchmark, a comprehensive simulation environment and dataset designed for end-to-end delivery tasks.
The Polar Research Station (PRS) Environment
The benchmark is built upon the Polar Research Station (PRS) Simulator, developed using Unity and the PhysX engine. This three-story environment includes diverse functional spaces such as laboratories, gyms, and offices, connected by a functional elevator system. Unlike many grid-based simulators, PRS utilizes continuous physics for more realistic interactions.
A defining feature of the PRS environment is the presence of ten autonomous Non-Player Characters (NPCs). These characters follow distinct daily routines driven by Large Language Models (LLMs), moving independently throughout the building. This creates a "moving target" problem, requiring the robot to not only locate an object but also actively search for the requester who may have changed locations since the initial instruction.
Methodology and Task Pipeline
The delivery task is framed as a multi-stage sequence: instruction parsing, object search, manipulation, navigation, and human delivery. The proposed baseline system utilizes a modular architecture to handle these stages.
Key technical components include:
- Instruction Parsing: An LLM (e.g., GPT-4) decomposes complex natural language into executable sub-tasks.
- Visual Scene Understanding: A Large Multimodal Model (LMM) provides common-sense reasoning, helping the robot identify likely "receptacles" (e.g., finding a bottle on a dining table) when an object is not immediately visible.
- Navigation and Manipulation: The system employs an A* algorithm for path planning and a 6-DOF grasping API for physical interaction within the continuous physics space.
Dataset and Evaluation
The PRS dataset comprises 13,296 language instructions, meticulously refined to ensure grammatical correctness and diversity. Unlike existing benchmarks such as ALFRED or Habitat, PRS is the first to integrate multi-floor navigation, autonomous NPCs, and human-centered task objectives simultaneously. Evaluation focuses on "zero-shot" reasoning, where models must generalize to the environment without specific prior training, mimicking the deployment conditions of commercial robots.
Experimental Results
Baseline tests using a combination of GPT-4o and Grounding DINO reveal the complexity of the benchmark. While the models excel at instruction parsing (71.7% success rate) and finding the target human (88.7%), the overall task success rate (SR) remains low at 32.2%.
| Metric | Success Rate (%) |
|---|---|
| Overall Task Success (SR) | 32.2 |
| Instruction Parsing | 71.7 |
| Manipulation (Grasping) | 59.2 |
| Human Search | 88.7 |
The primary bottlenecks are the initial object search and the physical challenges of manipulation, highlighting the difficulty of chaining long sequences of perception, reasoning, and execution in dynamic settings.
Conclusion and Future Directions
The Human-centered In-building Embodied Delivery Benchmark provides a rigorous testing ground for the next generation of delivery robots. By modeling the complexities of real-world buildings and human behavior, it exposes the limitations of current state-of-the-art models. Future work will focus on expanding architectural diversity, increasing NPC social complexity, and incorporating multi-modal human-robot interaction to further bridge the reality gap.
Summary updated