Abstract
Visual navigation in complex 3D environments remains a significant challenge for robotics, requiring agents to find specific objects based on visual and sensor data. While Large Language Models (LLMs) offer impressive reasoning capabilities, their application to navigation often suffers from brittle prompt engineering and a significant modality gap between text and sensor data. This paper presents a novel approach to fine-tuning Multimodal Large Language Models (MLLMs) for visual navigation. By integrating a dedicated history collector and training on human demonstrations with explicit collision signals, the proposed model achieves state-of-the-art performance on the Habitat-Matterport 3D (HM3D) dataset, outperforming traditional behavior cloning and reinforcement learning methods.
Introduction
Visual navigation, specifically Object Goal Navigation (ObjectNav), requires an agent to locate a specific object category (e.g., "bed," "toilet," "sofa") in an entirely unseen environment. Traditionally, this task has been approached using Reinforcement Learning (RL) or Imitation Learning (IL). Recently, researchers have attempted to leverage the reasoning power of Large Language Models (LLMs) by feeding them "massive text prompts" that describe the scene and the agent's history.
However, this prompt-centric approach faces two major hurdles. First, there is a distinct Modality Gap: pre-trained LLMs are optimized for text and do not natively understand visual observations, GPS coordinates, or compass data. Second, Prompt Fragility makes these systems difficult to scale, as performance is often highly sensitive to the specific wording of instructions and fails to capture low-level movement nuances. This research explores whether skipping "prompt gymnastics" in favor of directly fine-tuning an MLLM to process multimodal inputs can yield more robust navigation behavior.
Modular Architecture
The proposed system is built around a pre-trained LLaMA-13B model, augmented with specialized modules to handle multimodal inputs. The architecture consists of five core components designed to bridge the gap between raw sensor data and linguistic reasoning:
- Observation Encoding: Current camera images are processed using a Vision Transformer (ViT) and a Q-former (adapted from BLIP-2). This converts visual data into a set of visual tokens that the LLM can interpret as part of its input sequence.
- History Collector: To provide long-term context, a ResNet-50 paired with Gated Recurrent Units (GRUs) compresses the agent's entire journey—including past images, GPS coordinates, and previous actions—into a compact set of "history tokens."
- Projection Layers: Simple linear layers map these diverse visual and history features into the same embedding space as the LLM’s text tokens, ensuring a unified input representation.
- Simple Text Prompting: Instead of complex scene descriptions, the model uses a basic template: "Imagine you are a robot... please plan out your following action."
- Probability Distribution Output: Unlike models that output a single deterministic command, this architecture generates a probability distribution over all possible actions (Stop, Forward, Turn Left/Right, Look Up/Down). This allows the system to model uncertainty, leading to smoother path planning.
Training and Methodology
The model was trained using the Habitat-Matterport 3D (HM3D) dataset, utilizing 77,000 human demonstrations across 80 training scenes. A critical innovation in the training process is the use of collision signals. During the fine-tuning phase, the researchers combined ground-truth human actions with the outputs of existing behavior cloning models. Crucially, they "zeroed out" actions that would lead to collisions, effectively teaching the LLM that safety is a non-negotiable constraint. This explicit penalization significantly reduced the collision rate during testing from approximately 39.8% to 27.8%.
Experimental Results
The proposed method established a new state-of-the-art for ObjectNav on the HM3D dataset. Evaluation on 2,000 episodes across 20 validation scenes demonstrated significant improvements over both traditional and LLM-based baselines.
As shown in the results table, the model achieved a Success Rate of 67.90%, outperforming the previous best reinforcement learning fine-tuned (RL_Ft) method (66.15%) and standard imitation learning (59.80%). Furthermore, the SoftSPL (Path Efficiency) score of 0.3723 indicates that the agent not only finds the goal more often but does so using more efficient paths than its predecessors.
Key Insights
- Fine-tuning is Essential: Experiments with un-tuned LLMs (using MiniGPT-4) resulted in a 0% success rate, proving that zero-shot reasoning is insufficient for the low-level control required in navigation.
- The Power of History: The GRU-based history collector significantly outperformed a simple "sliding window" approach. Long-term context is vital for navigating complex layouts and avoiding repetitive "looping" behavior.
- Uncertainty Modeling: By outputting a probability distribution rather than a direct action, the model handles ambiguous situations more gracefully, resulting in fewer unnecessary turns and more stable trajectories.
Conclusion and Future Work
This research demonstrates that fine-tuning MLLMs with a robust history mechanism and safety-aware training can effectively solve complex visual navigation tasks without the need for exhaustive prompt engineering. While the results are promising, the authors note that future work could involve Multimodal Retrieval-Augmented Generation (RAG) to help agents remember distant parts of an environment. Additionally, exploring automated data generation through "frontier exploration" could help overcome the scarcity of expensive human demonstration data.
Summary updated