NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models

Gengze Zhou, Yicong Hong, Qi Wu

TL;DR

NavGPT is a novel navigation agent that uses the zero-shot reasoning power of Large Language Models (LLMs) like GPT-4 to solve Vision-and-Language Navigation (VLN) tasks. By translating visual observations into text, NavGPT demonstrates that LLMs can perform explicit high-level planning, landmark identification, and spatial reasoning, significantly outperforming previous non-trained methods.

What Problem Does This Paper Address?

Vision-and-Language Navigation (VLN) requires an embodied agent to follow natural language instructions (e.g., "Walk past the sink and stop between the refrigerator and dining table") in complex, unseen 3D environments. Traditionally, this requires massive amounts of task-specific training data to map pixels and text to actions.

However, these "black-box" models often struggle to generalize to new environments and offer little insight into why they make specific decisions. The authors of NavGPT ask: Can the emergent reasoning capabilities of LLMs be harnessed to create a universal, explainable navigation agent without any task-specific fine-tuning?

Methods

NavGPT operates as a purely LLM-based system that treats navigation as a sequence of text-based reasoning and action steps. The architecture, shown in Figure 1, consists of several key components:

  • Visual Perceptron: Since LLMs are text-based, NavGPT uses Visual Foundation Models (VFMs) as "translators." It employs BLIP-2 to generate natural language descriptions of the scene and Faster R-CNN to detect objects and their relative distances.
  • Prompt Manager: This component synthesizes the instruction, the current visual description, and the navigation history into a structured prompt. It follows the ReAct (Reason + Act) paradigm, forcing the LLM to generate a "Thought" (explicit reasoning) before choosing an "Action" (moving to a specific viewpoint).
  • History Buffer & Summarizer: To handle long trajectories without exceeding the LLM's context window, NavGPT uses a GPT-3.5-based summarizer to compress past observations into concise historical summaries.
  • Zero-Shot Execution: Crucially, the model is not trained on any VLN data. It relies entirely on the pre-trained knowledge and reasoning abilities of GPT-4.
Figure 1: The architecture of NavGPT. NavGPT synergizes reasoning and actions in LLMs to perform zero-shot Vision-and-Language Navigation following navigation system principles. It interactives with different visual foundation models to adapt multi-modality inputs, handle the length of history with a history buffer and a GPT-3.5 summarizer, and aggregate various sources of information through a prompt manager. NavGPT parse the generated results from LLMs (LLM Thoughts and LLM Action) to move to the next viewpoint.

Datasets

The authors evaluated NavGPT on the R2R (Room-to-Room) dataset, a standard benchmark for VLN. R2R contains over 7,000 trajectories in indoor environments simulated via the Matterport3D platform. The experiments specifically focused on the "val unseen" split, testing the agent's ability to navigate in buildings it had never "seen" during the LLM's initial training.

Key Insights and Ideas

  • Explicit High-Level Planning: Unlike traditional models, NavGPT can explicitly decompose complex instructions into sub-goals. For example, it might "think" that it needs to find a hallway first before looking for a specific room.
  • The "Textual Bottleneck": The primary limitation isn't the LLM's intelligence, but the loss of information when converting 3D visual scenes into a few sentences of text. If a VFM fails to mention a small landmark, the LLM has no way of knowing it exists.
  • Spatial and Historical Awareness: GPT-4 shows a surprising ability to maintain a mental map. The authors demonstrated that GPT-4 could take a text-based navigation history and accurately draw a top-down metric trajectory (Figure 4) or generate new, high-quality instructions for the path it just took.
  • Exception Handling: NavGPT can use commonsense reasoning to adjust plans. If it expects a bathroom but sees a kitchen, it can reason that it might have taken a wrong turn and needs to backtrack or explore adjacent areas.
Figure 4: We evaluate GPT-4 on a case where NavGPT successfully follows the ground truth path, using only the historical actions Aăt`1 and observations Oăt`1 to generate an instruction (without reasoning trace Răt`1 to avoid information leaking), and using the entire navigation history Hăt`1 to draw a top-down trajectory.

Key Results

NavGPT sets a new state-of-the-art for zero-shot (no training) navigation:

  • Success Rate (SR): NavGPT (using GPT-4) achieved a 34% Success Rate on unseen environments. While this is lower than fine-tuned models (which reach ~60-70%), it is a massive leap over previous non-trained baselines like DuET, which scored only 1%.
  • Navigation Error (NE): NavGPT maintained a mean error of 6.46 meters, significantly better than the 9.74 meters of other non-trained approaches.
  • Ablation Studies: The researchers found that using a 45-degree Field of View (FoV) for scene descriptions and including depth information were critical for performance, increasing the Success Rate by nearly 5% and 0.7% respectively.
Table 1: Comparison with previous methods on R2R validation unseen split.

Limitations and Future Work

The authors identify two main hurdles:

  1. Information Degradation: Translating rich visual data into static text descriptions is lossy. Future work should explore Multi-modal LLMs that can process images directly as tokens.
  2. Memory Decay: Summarizing history to save space inevitably leads to the loss of fine-grained details needed for precise backtracking.

The paper concludes that while LLMs aren't yet ready to replace specialized navigation models, their high-level planning and reasoning should be integrated into future hybrid systems to create more robust and explainable embodied agents.

Summary updated

Abstract

Trained with an unprecedented scale of data, large language models (LLMs) like ChatGPT and GPT-4 exhibit the emergence of significant reasoning abilities from model scaling. Such a trend underscored the potential of training LLMs with unlimited language data, advancing the development of a universal embodied agent. In this work, we introduce the NavGPT, a purely LLM-based instruction-following navigation agent, to reveal the reasoning capability of GPT models in complex embodied scenes by performing zero-shot sequential action prediction for vision-and-language navigation (VLN). At each step, NavGPT takes the textual descriptions of visual observations, navigation history, and future explorable directions as inputs to reason the agent's current status, and makes the decision to approach the target. Through comprehensive experiments, we demonstrate NavGPT can explicitly perform high-level planning for navigation, including decomposing instruction into sub-goal, integrating commonsense knowledge relevant to navigation task resolution, identifying landmarks from observed scenes, tracking navigation progress, and adapting to exceptions with plan adjustment. Furthermore, we show that LLMs is capable of generating high-quality navigational instructions from observations and actions along a path, as well as drawing accurate top-down metric trajectory given the agent's navigation history. Despite the performance of using NavGPT to zero-shot R2R tasks still falling short of trained models, we suggest adapting multi-modality inputs for LLMs to use as visual navigation agents and applying the explicit reasoning of LLMs to benefit learning-based models.
BibTeX
@inproceedings{zhou2024navgpt,
  author = {Gengze Zhou and Yicong Hong and Qi Wu},
  title = {{NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i7.28597},
}
Figure