Abstract
Open-Vocabulary Mobile Manipulation (OVMM) is a crucial capability for autonomous robots, especially when faced with the challenges posed by unknown and dynamic environments. This task requires robots to explore and build a semantic understanding of their surroundings, generate feasible plans to achieve manipulation goals, adapt to environmental changes, and comprehend natural language instructions from humans. To address these challenges, we propose a novel framework that leverages the zero-shot detection and grounded recognition capabilities of pretraining visual-language models (VLMs) combined with dense 3D entity reconstruction to build 3D semantic maps. Additionally, we utilize large language models (LLMs) for spatial region abstraction and online planning, incorporating human instructions and spatial semantic context. We have built a 10-DoF mobile manipulation robotic platform JSR-1 and demonstrated in real-world robot experiments that our proposed framework can effectively capture spatial semantics and process natural language user instructions for zero-shot OVMM tasks under dynamic environment settings, with an overall navigation and task success rate of 80.95% and 73.33% over 105 episodes, and better SFT and SPL by 157.18% and 19.53% respectively compared to the baseline. Furthermore, the framework is capable of replanning towards the next most probable candidate location based on the spatial semantic context derived from the 3D semantic map when initial plans fail, keeping an average success rate of 76.67%.
Introduction
Open-Vocabulary Mobile Manipulation (OVMM) represents a significant milestone in service robotics, requiring a robot to locate and manipulate arbitrary objects within mobile environments. Achieving this in the real world involves overcoming three primary hurdles: navigating unseen environments without pre-built maps, adapting to dynamic changes where objects are moved, and resolving natural language ambiguities or incorrect human hints. This research introduces a two-stage framework that enables the JSR-1 robot to "think" through these contradictions using semantic common sense, achieving high success rates in zero-shot tasks without task-specific training.
Methodology
The proposed framework operates in two distinct stages, integrating 3D semantic mapping with the reasoning capabilities of Large Language Models (LLMs) and Vision-Language Models (VLMs).
Stage 1: 3D Semantic Mapping (3DSMap)
The robot explores the environment using heuristic algorithms to build a multi-layered 3D semantic map. This map consists of three layers:
- Structural Layer: Built using SLAM to capture the physical 3D geometry.
- Instance Semantic Layer: Utilizes VLMs (Grounded SAM and InternVL) to detect and segment specific objects in real-time.
- Abstract Region Semantic Layer: Employs an LLM (GPT-4o) to perform region abstraction, clustering objects into functional areas like an "Office" or "Kitchen."
Stage 2: Semantics-aware Manipulation
Upon receiving a natural language instruction, the system parses the target object and any location hints. The LLM prioritizes search areas based on semantic relevance (e.g., a controller is likely in an "Entertainment Area"). If the object is not found at the initial location due to environmental changes, the robot leverages the 3D semantic map to autonomously re-plan and navigate to the next most probable region.
Experimental Setup
The framework was evaluated in a 200 m² indoor space featuring five functional regions (Entertainment, Washing, Cooking, Bar, Office) and 20 object categories. The benchmark consisted of 135 independent real-world episodes categorized into five experimental groups: NoHint, Random, Hinting, ErrantSemantics, and Misleading. These groups tested the robot's ability to handle various levels of instruction clarity and accuracy.
Results and Discussion
The framework demonstrated substantial improvements over a random search baseline across all key metrics. The system achieved an overall task success rate of 73.33% and a navigation success rate (SN) of 80.95%.
Key performance indicators include:
- Efficiency: The Success weighted by Path Length (SPL) improved by 19.53% over the baseline.
- Search Speed: The Semantic Find Time (SFT) showed a 157.18% increase in efficiency.
- Robustness: In the "Misleading" group, where robots were given incorrect location hints, the system maintained a 66.67% success rate by autonomously correcting its path based on semantic common sense.
Conclusion
This work demonstrates that combining foundation models with dense 3D semantic mapping allows mobile robots to navigate complex, dynamic environments with human-like reasoning. Future work will focus on improving exploration efficiency through advanced autonomous mapping, scaling the system for multi-floor environments, and implementing multi-agent collaboration for real-time map updates in highly fluid spaces.
Summary updated