Multimodal Large Language Model for Visual Navigation

Yao-Hung Hubert Tsai, Vansh Dhar, Jialu Li, Bowen Zhang, Jian ZhangarXiv 2023 · DOI · arXiv

Abstract

Visual navigation in complex 3D environments remains a significant challenge for robotics, requiring agents to find specific objects based on visual and sensor data. While Large Language Models (LLMs) offer impressive reasoning capabilities, their application to navigation often suffers from brittle prompt engineering and a significant modality gap between text and sensor data. This paper presents a novel approach to fine-tuning Multimodal Large Language Models (MLLMs) for visual navigation. By integrating a dedicated history collector and training on human demonstrations with explicit collision signals, the proposed model achieves state-of-the-art performance on the Habitat-Matterport 3D (HM3D) dataset, outperforming traditional behavior cloning and reinforcement learning methods.

Introduction

Visual navigation, specifically Object Goal Navigation (ObjectNav), requires an agent to locate a specific object category (e.g., "bed," "toilet," "sofa") in an entirely unseen environment. Traditionally, this task has been approached using Reinforcement Learning (RL) or Imitation Learning (IL). Recently, researchers have attempted to leverage the reasoning power of Large Language Models (LLMs) by feeding them "massive text prompts" that describe the scene and the agent's history.

However, this prompt-centric approach faces two major hurdles. First, there is a distinct Modality Gap: pre-trained LLMs are optimized for text and do not natively understand visual observations, GPS coordinates, or compass data. Second, Prompt Fragility makes these systems difficult to scale, as performance is often highly sensitive to the specific wording of instructions and fails to capture low-level movement nuances. This research explores whether skipping "prompt gymnastics" in favor of directly fine-tuning an MLLM to process multimodal inputs can yield more robust navigation behavior.

Modular Architecture

The proposed system is built around a pre-trained LLaMA-13B model, augmented with specialized modules to handle multimodal inputs. The architecture consists of five core components designed to bridge the gap between raw sensor data and linguistic reasoning:

  • Observation Encoding: Current camera images are processed using a Vision Transformer (ViT) and a Q-former (adapted from BLIP-2). This converts visual data into a set of visual tokens that the LLM can interpret as part of its input sequence.
  • History Collector: To provide long-term context, a ResNet-50 paired with Gated Recurrent Units (GRUs) compresses the agent's entire journey—including past images, GPS coordinates, and previous actions—into a compact set of "history tokens."
  • Projection Layers: Simple linear layers map these diverse visual and history features into the same embedding space as the LLM’s text tokens, ensuring a unified input representation.
  • Simple Text Prompting: Instead of complex scene descriptions, the model uses a basic template: "Imagine you are a robot... please plan out your following action."
  • Probability Distribution Output: Unlike models that output a single deterministic command, this architecture generates a probability distribution over all possible actions (Stop, Forward, Turn Left/Right, Look Up/Down). This allows the system to model uncertainty, leading to smoother path planning.
The proposed modular architecture integrating observation encoding, history collection, and a pre-trained LLM.

Training and Methodology

The model was trained using the Habitat-Matterport 3D (HM3D) dataset, utilizing 77,000 human demonstrations across 80 training scenes. A critical innovation in the training process is the use of collision signals. During the fine-tuning phase, the researchers combined ground-truth human actions with the outputs of existing behavior cloning models. Crucially, they "zeroed out" actions that would lead to collisions, effectively teaching the LLM that safety is a non-negotiable constraint. This explicit penalization significantly reduced the collision rate during testing from approximately 39.8% to 27.8%.

Experimental Results

The proposed method established a new state-of-the-art for ObjectNav on the HM3D dataset. Evaluation on 2,000 episodes across 20 validation scenes demonstrated significant improvements over both traditional and LLM-based baselines.

Comparison of the proposed method against various reinforcement learning and behavior cloning baselines.

As shown in the results table, the model achieved a Success Rate of 67.90%, outperforming the previous best reinforcement learning fine-tuned (RL_Ft) method (66.15%) and standard imitation learning (59.80%). Furthermore, the SoftSPL (Path Efficiency) score of 0.3723 indicates that the agent not only finds the goal more often but does so using more efficient paths than its predecessors.

Key Insights

  • Fine-tuning is Essential: Experiments with un-tuned LLMs (using MiniGPT-4) resulted in a 0% success rate, proving that zero-shot reasoning is insufficient for the low-level control required in navigation.
  • The Power of History: The GRU-based history collector significantly outperformed a simple "sliding window" approach. Long-term context is vital for navigating complex layouts and avoiding repetitive "looping" behavior.
  • Uncertainty Modeling: By outputting a probability distribution rather than a direct action, the model handles ambiguous situations more gracefully, resulting in fewer unnecessary turns and more stable trajectories.

Conclusion and Future Work

This research demonstrates that fine-tuning MLLMs with a robust history mechanism and safety-aware training can effectively solve complex visual navigation tasks without the need for exhaustive prompt engineering. While the results are promising, the authors note that future work could involve Multimodal Retrieval-Augmented Generation (RAG) to help agents remember distant parts of an environment. Additionally, exploring automated data generation through "frontier exploration" could help overcome the scarcity of expensive human demonstration data.

Summary updated

Abstract

Recent efforts to enable visual navigation using large language models have mainly focused on developing complex prompt systems. These systems incorporate instructions, observations, and history into massive text prompts, which are then combined with pre-trained large language models to facilitate visual navigation. In contrast, our approach aims to fine-tune large language models for visual navigation without extensive prompt engineering. Our design involves a simple text prompt, current observations, and a history collector model that gathers information from previous observations as input. For output, our design provides a probability distribution of possible actions that the agent can take during navigation. We train our model using human demonstrations and collision signals from the Habitat-Matterport 3D Dataset (HM3D). Experimental results demonstrate that our method outperforms state-of-the-art behavior cloning methods and effectively reduces collision rates.
BibTeX
@misc{tsai2023multimodal,
  author = {Yao-Hung Hubert Tsai and Vansh Dhar and Jialu Li and Bowen Zhang and Jian Zhang},
  title = {{Multimodal Large Language Model for Visual Navigation}},
  year = {2023},
  eprint = {2310.08669},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.08669},
}
Figure