TL;DR
Navi2Gaze is a new framework that enables robots to navigate toward objects with the correct orientation—or "gaze"—required for specific tasks, such as facing a refrigerator to open it. By using Vision-Language Models (VLMs) like GPT-4V to score potential standing positions based on common-sense reasoning, the system achieves a 68.8% reduction in distance-to-goal errors compared to existing state-of-the-art methods.
What Problem Does This Paper Address?
In the field of Zero-Shot Object Navigation (ZSON), success is typically defined by a robot simply arriving near a target. However, for a home assistant robot, just being "near" isn't enough. If a robot is asked to "open the refrigerator" but ends up facing the back or the side of the appliance, it cannot perform the next step.
Existing navigation methods often overlook the robot's final pose (position and orientation). While Vision-Language Models (VLMs) possess the "common sense" to know that humans stand in front of a fridge to open it, they lack the 3D spatial reasoning to plan a path there directly. Navi2Gaze bridges this "motion gap" by translating the linguistic common sense of VLMs into precise, task-aware spatial coordinates.
Methods
The authors propose a VLM-driven, coarse-to-fine scoring mechanism that functions as the robot's "decision-making brain." The workflow consists of four main stages:
- Target Scene Identification: The robot uses a VLM to scan a set of captured images and identify which one provides the clearest view of the target object.
- Task-Aware Space Reconstruction: Once the target is found, the robot performs a "scanning dance"—rotating and tilting its camera—to collect dense RGB-D data. This is used to build a local 3D point cloud of the object and its surroundings.
- Location Candidate Generation: The system projects the 3D data onto a 2D ground map. It then generates a set of "candidate regions" (potential standing spots) around the object, filtering out areas blocked by obstacles.
- Sequential Decision Making: This is the core innovation. The robot overlays numbered markers on its current view (a technique called "Visual Prompting" or "Set-of-Mark"). It feeds these images to the VLM, asking it to score which marker represents the best spot to perform the task (e.g., "Which position is best for opening this drawer?"). The robot then navigates to the highest-scoring spot and rotates to face the target.
Crucially, this method does not require a pre-built 3D semantic map of the entire environment, making it highly flexible for new homes.
Datasets
The researchers evaluated Navi2Gaze using both simulation and real-world hardware:
- Simulation: They used the Habitat simulator with the HM3D (Habitat-Matterport 3D) v0.2 dataset, which contains high-resolution, photorealistic 3D scans of real indoor environments. They tested 68 different tasks across nine scenes.
- Real-World: Experiments were conducted using a custom-built mobile manipulation robot equipped with an Intel RealSense RGB-D camera, LiDAR, and a 6-axis robotic arm.
Key Insights and Ideas
- Common Sense as a Heuristic: The paper treats VLM knowledge as a navigation heuristic. Instead of asking the VLM to "drive," they ask it to "judge" which of several valid physical locations makes the most sense for a human-like task.
- The Importance of "Gazing": The authors redefine the goal of navigation from "proximity" to "alignment." They argue that "Distance to Goal" (DTG) should be measured relative to the optimal operation position, not just the object's center.
- Visual Prompting Overcomes Spatial Limits: By drawing numbered circles on a 2D image and asking the VLM to pick a number, they bypass the VLM's inability to output precise 3D coordinates.
- Coarse-to-Fine Strategy: By first picking the best general scene and then picking the specific spot within that scene, the system minimizes computational waste and improves accuracy.
Key Results
Navi2Gaze was compared against several baselines, including VLMaps, L3MVN, and SemExp.
- Precision: In simulations, Navi2Gaze reduced the Distance to Goal (DTG) by 68.8% compared to VLMaps.
- Success Rate (SR): When robots were intentionally started at "bad" angles (facing the back or side of an object), Navi2Gaze showed a 61.9% increase in success rate over VLMaps, proving its ability to reposition itself intelligently.
- Qualitative Success: Real-world tests showed the robot successfully navigating to and correctly facing complex targets like water dispensers, fridges, and drawers—tasks where orientation is mandatory for success.
Limitations and Future Work
- Segmentation Dependency: The system relies heavily on segmentation models (like SAM or SEEM) to identify objects. If these models fail due to poor lighting or unusual object shapes, the entire pipeline can break.
- Depth Noise: RGB-D cameras often provide "noisy" depth data, which can lead to inaccuracies in the 3D point cloud reconstruction and, consequently, the placement of candidate regions.
- Inference Speed: Querying a large VLM like GPT-4V multiple times per task introduces latency. Future work might involve using smaller, locally-hosted VLMs to speed up the decision-making process.
- Dynamic Environments: The current model assumes a static scene during the "reconstruction" phase; handling moving people or changing obstacles remains a challenge.
Summary updated