Affordance-Guided Coarse-to-Fine Exploration for Base Placement in Open-Vocabulary Mobile Manipulation

Tzu-Jung Lin, Jia-Fong Yeh, Hung-Ting Su, Chung-Yi Lin, Yi-Ting Chen, Winston H. Hsu

Abstract

In Open-Vocabulary Mobile Manipulation (OVMM), a robot's success often hinges on its initial base placement. This paper presents a zero-shot framework that integrates high-level semantic reasoning from Vision-Language Models (VLMs) with iterative geometric optimization. By projecting affordance cues into a navigation-friendly space and employing a coarse-to-fine search strategy, the system achieves an 85% success rate across complex tasks such as opening articulated furniture and grasping objects in cluttered environments.

Introduction: The Challenge of Base Placement

Modern robots excel at navigating toward objects, but the "final meter" of a task—positioning the base for manipulation—remains a significant hurdle. In OVMM, robots must interpret natural language commands like "Open the cabinet" in novel settings. Failure often occurs not because the robot cannot see the object, but because it parks in a configuration that is either geometrically unreachable or semantically incorrect.

Failure to reach a pot handle due to poor base placement.

The authors identify two primary bottlenecks:

  1. Joint Reasoning: Balancing semantic intent (e.g., facing a handle) with geometric feasibility (e.g., avoiding collisions).
  2. Perceptual Limitations: Standard VLMs struggle with objects outside their immediate field of view or with fine-grained spatial precision required for reachability.
Failure to open a cabinet due to incorrect orientation relative to the drawer.

Methodology

The proposed Affordance-Guided Coarse-to-Fine Exploration addresses these issues through a two-stage process.

Affordance Guidance Projection

To bridge the gap between 3D visual perception and 2D navigation, the system generates "cross-modal" representations. It overlays color-coded directional arrows on both the egocentric RGB image ($I_{aff}$) and the top-down obstacle map ($M^+_{local}$). This "spatial Rosetta Stone" allows the VLM to map visual affordances directly to navigation coordinates, enabling reasoning beyond the robot's immediate camera frame.

Coarse-to-Fine Optimization

The robot iteratively searches for the optimal base placement $x$ over several steps. The search is guided by a scoring function $w(x)$ that balances semantic and geometric weights: $$w(x) = w_{geo}(x)^{\alpha_t} \cdot w_{sem}(x)^{1-\alpha_t}$$ A sigmoid schedule for $\alpha_t$ ensures that the robot first prioritizes semantics (finding the correct side of an object) before refining its position based on geometric precision (achieving the ideal distance for arm reach).

Evaluation and Results

The framework was tested in NVIDIA Isaac Sim using a TIAGo++ manipulator on five diverse tasks, including articulated object manipulation (cabinets, dishwashers) and pick-and-place operations.

The system achieved an 85% overall success rate, significantly outperforming baselines like PIVOT and standard geometric planners. Notably, on the "Open Dishwasher" task—which requires precise approach angles to avoid the opening door—the method succeeded in 17 out of 20 trials, compared to only 5 for traditional planners. Ablation studies confirmed that the affordance guidance markers were critical, as their removal dropped success rates to 48%.

Conclusion

By leveraging VLMs for coarse semantic guidance and geometric optimization for fine-grained precision, this framework provides a robust solution for zero-shot base placement. Future work will focus on integrating full arm trajectory checking and addressing sensor noise inherent in real-world hardware deployments.

Summary updated

BibTeX
@inproceedings{lin2026affordance,
  author = {Tzu-Jung Lin and Jia-Fong Yeh and Hung-Ting Su and Chung-Yi Lin and Yi-Ting Chen and Winston H. Hsu},
  title = {{Affordance-Guided Coarse-to-Fine Exploration for Base Placement in Open-Vocabulary Mobile Manipulation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i22.38909},
}
Figure