Abstract: Task-aware navigation continues to be a challenging area of research, especially in scenarios involving open vocabulary. Previous studies primarily focus on finding suitable locations for task completion, often overlooking the importance of the robot's pose. However, the robot's orientation is crucial for successfully completing tasks because of how objects are arranged (e.g., to open a refrigerator door). Humans intuitively navigate to objects with the right orientation using semantics and common sense. For instance, when opening a refrigerator, we naturally stand in front of it rather than to the side. Recent advances suggest that Vision-Language Models (VLMs) can provide robots with similar common sense. Therefore, we develop a VLM-driven method called Navigation-to-Gaze (Navi2Gaze) for efficient navigation and object gazing based on task descriptions. This method uses the VLM to score and select the best pose from numerous candidates automatically. In evaluations on multiple photorealistic simulation benchmarks, Navi2Gaze significantly outperforms existing approaches by precisely determining the optimal orientation relative to target objects, resulting in a 68.8% reduction in Distance to Goal (DTG).
Introduction
In the field of Zero-Shot Object Navigation (ZSON), success is typically defined by a robot simply arriving near a target. However, for a home assistant robot, just being "near" isn't enough. If a robot is asked to "open the refrigerator" but ends up facing the back or the side of the appliance, it cannot perform the next step. Existing navigation methods often overlook the robot's final pose (position and orientation). While Vision-Language Models (VLMs) possess the "common sense" to know that humans stand in front of a fridge to open it, they lack the 3D spatial reasoning to plan a path there directly. Navi2Gaze bridges this "motion gap" by translating the linguistic common sense of VLMs into precise, task-aware spatial coordinates.
Methodology
The Navi2Gaze framework utilizes a VLM-driven, coarse-to-fine scoring mechanism that functions as the robot's decision-making core. The workflow consists of four main stages:
- Target Scene Identification: The robot uses a VLM to scan a set of captured images and identify which one provides the clearest view of the target object.
- Task-Aware Space Reconstruction: Once the target is found, the robot performs a "scanning dance" to collect dense RGB-D data. This is used to build a local 3D point cloud of the object and its surroundings.
- Location Candidate Generation: The system projects the 3D data onto a 2D ground map. It then generates a set of "candidate regions" (potential standing spots) around the object, filtering out areas blocked by obstacles.
- Sequential Decision Making: This is the core innovation. The robot overlays numbered markers on its current view (Visual Prompting). It feeds these images to the VLM, asking it to score which marker represents the best spot to perform the task. The robot then navigates to the highest-scoring spot and rotates to face the target.
Experimental Setup
The researchers evaluated Navi2Gaze using both simulation and real-world hardware:
- Simulation: The Habitat simulator was used with the HM3D (Habitat-Matterport 3D) v0.2 dataset, which contains high-resolution, photorealistic 3D scans of real indoor environments. The team tested 68 different tasks across nine scenes.
- Real-World: Experiments were conducted using a custom-built mobile manipulation robot equipped with an Intel RealSense RGB-D camera, LiDAR, and a 6-axis robotic arm.
Results
Navi2Gaze was compared against several baselines, including VLMaps, L3MVN, and SemExp.
- Precision: In simulations, Navi2Gaze reduced the Distance to Goal (DTG) by 68.8% compared to VLMaps.
- Success Rate (SR): When robots were intentionally started at "bad" angles (facing the back or side of an object), Navi2Gaze showed a 61.9% increase in success rate over VLMaps, proving its ability to reposition itself intelligently.
- Qualitative Success: Real-world tests showed the robot successfully navigating to and correctly facing complex targets like water dispensers, fridges, and drawers—tasks where orientation is mandatory for success.
Discussion and Limitations
While Navi2Gaze significantly improves task-aware navigation, several challenges remain. The system relies heavily on segmentation models (like SAM or SEEM); failures in these models due to lighting or object shape can disrupt the pipeline. Additionally, RGB-D cameras often provide noisy depth data, affecting 3D reconstruction. Inference latency from large VLMs like GPT-4V is also a factor, suggesting future work should explore smaller, locally-hosted models. Finally, the current model assumes a static scene; handling dynamic environments with moving obstacles remains a subject for future research.
Summary updated