Abstract
In Open-Vocabulary Mobile Manipulation (OVMM), a robot's success often hinges on its initial base placement. This paper presents a zero-shot framework that integrates high-level semantic reasoning from Vision-Language Models (VLMs) with iterative geometric optimization. By projecting affordance cues into a navigation-friendly space and employing a coarse-to-fine search strategy, the system achieves an 85% success rate across complex tasks such as opening articulated furniture and grasping objects in cluttered environments.
Introduction: The Challenge of Base Placement
Modern robots excel at navigating toward objects, but the "final meter" of a task—positioning the base for manipulation—remains a significant hurdle. In OVMM, robots must interpret natural language commands like "Open the cabinet" in novel settings. Failure often occurs not because the robot cannot see the object, but because it parks in a configuration that is either geometrically unreachable or semantically incorrect.
The authors identify two primary bottlenecks:
- Joint Reasoning: Balancing semantic intent (e.g., facing a handle) with geometric feasibility (e.g., avoiding collisions).
- Perceptual Limitations: Standard VLMs struggle with objects outside their immediate field of view or with fine-grained spatial precision required for reachability.
Methodology
The proposed Affordance-Guided Coarse-to-Fine Exploration addresses these issues through a two-stage process.
Affordance Guidance Projection
To bridge the gap between 3D visual perception and 2D navigation, the system generates "cross-modal" representations. It overlays color-coded directional arrows on both the egocentric RGB image ($I_{aff}$) and the top-down obstacle map ($M^+_{local}$). This "spatial Rosetta Stone" allows the VLM to map visual affordances directly to navigation coordinates, enabling reasoning beyond the robot's immediate camera frame.
Coarse-to-Fine Optimization
The robot iteratively searches for the optimal base placement $x$ over several steps. The search is guided by a scoring function $w(x)$ that balances semantic and geometric weights: $$w(x) = w_{geo}(x)^{\alpha_t} \cdot w_{sem}(x)^{1-\alpha_t}$$ A sigmoid schedule for $\alpha_t$ ensures that the robot first prioritizes semantics (finding the correct side of an object) before refining its position based on geometric precision (achieving the ideal distance for arm reach).
Evaluation and Results
The framework was tested in NVIDIA Isaac Sim using a TIAGo++ manipulator on five diverse tasks, including articulated object manipulation (cabinets, dishwashers) and pick-and-place operations.
The system achieved an 85% overall success rate, significantly outperforming baselines like PIVOT and standard geometric planners. Notably, on the "Open Dishwasher" task—which requires precise approach angles to avoid the opening door—the method succeeded in 17 out of 20 trials, compared to only 5 for traditional planners. Ablation studies confirmed that the affordance guidance markers were critical, as their removal dropped success rates to 48%.
Conclusion
By leveraging VLMs for coarse semantic guidance and geometric optimization for fine-grained precision, this framework provides a robust solution for zero-shot base placement. Future work will focus on integrating full arm trajectory checking and addressing sensor noise inherent in real-world hardware deployments.
Summary updated