Navi2Gaze: Leveraging Foundation Models for Navigation and Target Gazing

Jun Zhu, Zihao Du, Haotian Xu, Fengbo Lan, Zilong Zheng, Bo Ma, Shengjie Wang, Tao Zhang

TL;DR

Navi2Gaze is a new framework that enables robots to navigate toward objects with the correct orientation—or "gaze"—required for specific tasks, such as facing a refrigerator to open it. By using Vision-Language Models (VLMs) like GPT-4V to score potential standing positions based on common-sense reasoning, the system achieves a 68.8% reduction in distance-to-goal errors compared to existing state-of-the-art methods.

What Problem Does This Paper Address?

In the field of Zero-Shot Object Navigation (ZSON), success is typically defined by a robot simply arriving near a target. However, for a home assistant robot, just being "near" isn't enough. If a robot is asked to "open the refrigerator" but ends up facing the back or the side of the appliance, it cannot perform the next step.

Existing navigation methods often overlook the robot's final pose (position and orientation). While Vision-Language Models (VLMs) possess the "common sense" to know that humans stand in front of a fridge to open it, they lack the 3D spatial reasoning to plan a path there directly. Navi2Gaze bridges this "motion gap" by translating the linguistic common sense of VLMs into precise, task-aware spatial coordinates.

Methods

The authors propose a VLM-driven, coarse-to-fine scoring mechanism that functions as the robot's "decision-making brain." The workflow consists of four main stages:

  1. Target Scene Identification: The robot uses a VLM to scan a set of captured images and identify which one provides the clearest view of the target object.
  2. Task-Aware Space Reconstruction: Once the target is found, the robot performs a "scanning dance"—rotating and tilting its camera—to collect dense RGB-D data. This is used to build a local 3D point cloud of the object and its surroundings.
  3. Location Candidate Generation: The system projects the 3D data onto a 2D ground map. It then generates a set of "candidate regions" (potential standing spots) around the object, filtering out areas blocked by obstacles.
  4. Sequential Decision Making: This is the core innovation. The robot overlays numbered markers on its current view (a technique called "Visual Prompting" or "Set-of-Mark"). It feeds these images to the VLM, asking it to score which marker represents the best spot to perform the task (e.g., "Which position is best for opening this drawer?"). The robot then navigates to the highest-scoring spot and rotates to face the target.

Crucially, this method does not require a pre-built 3D semantic map of the entire environment, making it highly flexible for new homes.

Datasets

The researchers evaluated Navi2Gaze using both simulation and real-world hardware:

  • Simulation: They used the Habitat simulator with the HM3D (Habitat-Matterport 3D) v0.2 dataset, which contains high-resolution, photorealistic 3D scans of real indoor environments. They tested 68 different tasks across nine scenes.
  • Real-World: Experiments were conducted using a custom-built mobile manipulation robot equipped with an Intel RealSense RGB-D camera, LiDAR, and a 6-axis robotic arm.

Key Insights and Ideas

  • Common Sense as a Heuristic: The paper treats VLM knowledge as a navigation heuristic. Instead of asking the VLM to "drive," they ask it to "judge" which of several valid physical locations makes the most sense for a human-like task.
  • The Importance of "Gazing": The authors redefine the goal of navigation from "proximity" to "alignment." They argue that "Distance to Goal" (DTG) should be measured relative to the optimal operation position, not just the object's center.
  • Visual Prompting Overcomes Spatial Limits: By drawing numbered circles on a 2D image and asking the VLM to pick a number, they bypass the VLM's inability to output precise 3D coordinates.
  • Coarse-to-Fine Strategy: By first picking the best general scene and then picking the specific spot within that scene, the system minimizes computational waste and improves accuracy.

Key Results

Navi2Gaze was compared against several baselines, including VLMaps, L3MVN, and SemExp.

  • Precision: In simulations, Navi2Gaze reduced the Distance to Goal (DTG) by 68.8% compared to VLMaps.
  • Success Rate (SR): When robots were intentionally started at "bad" angles (facing the back or side of an object), Navi2Gaze showed a 61.9% increase in success rate over VLMaps, proving its ability to reposition itself intelligently.
  • Qualitative Success: Real-world tests showed the robot successfully navigating to and correctly facing complex targets like water dispensers, fridges, and drawers—tasks where orientation is mandatory for success.

Limitations and Future Work

  • Segmentation Dependency: The system relies heavily on segmentation models (like SAM or SEEM) to identify objects. If these models fail due to poor lighting or unusual object shapes, the entire pipeline can break.
  • Depth Noise: RGB-D cameras often provide "noisy" depth data, which can lead to inaccuracies in the 3D point cloud reconstruction and, consequently, the placement of candidate regions.
  • Inference Speed: Querying a large VLM like GPT-4V multiple times per task introduces latency. Future work might involve using smaller, locally-hosted VLMs to speed up the decision-making process.
  • Dynamic Environments: The current model assumes a static scene during the "reconstruction" phase; handling moving people or changing obstacles remains a challenge.

Summary updated

Abstract

Task-aware navigation continues to be a challenging area of research, especially in scenarios involving open vocabulary. Previous studies primarily focus on finding suitable locations for task completion, often overlooking the importance of the robot's pose. However, the robot's orientation is crucial for successfully completing tasks because of how objects are arranged (e.g., to open a refrigerator door). Humans intuitively navigate to objects with the right orientation using semantics and common sense. For instance, when opening a refrigerator, we naturally stand in front of it rather than to the side. Recent advances suggest that Vision-Language Models (VLMs) can provide robots with similar common sense. Therefore, we develop a VLM-driven method called Navigation-to-Gaze (Navi2Gaze) for efficient navigation and object gazing based on task descriptions. This method uses the VLM to score and select the best pose from numerous candidates automatically. In evaluations on multiple photorealistic simulation benchmarks, Navi2Gaze significantly outperforms existing approaches by precisely determining the optimal orientation relative to target objects, resulting in a 68.8% reduction in Distance to Goal (DTG). Real-world video demonstrations can be found on the supplementary website
BibTeX
@inproceedings{zhu2025navi,
  author = {Jun Zhu and Zihao Du and Haotian Xu and Fengbo Lan and Zilong Zheng and Bo Ma and Shengjie Wang and Tao Zhang},
  title = {{Navi2Gaze: Leveraging Foundation Models for Navigation and Target Gazing}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11247165},
}

Loading related content...

Figure