Abstract
Natural language instructions for visual navigation often use scene descriptions (e.g., "bedroom") and object references (e.g., "green chairs") to provide a breadcrumb trail to a goal location. This work presents a transformer-based vision-and-language navigation (VLN) agent that uses two different visual encoders -- a scene classification network and an object detector -- which produce features that match these two distinct types of visual cues. In our method, scene features contribute high-level contextual information that supports object-level processing. With this design, our model is able to use vision-and-language pretraining to substantially improve performance on the Room-to-Room (R2R) and Room-Across-Room (RxR) benchmarks. Specifically, our approach leads to improvements of 1.8% absolute in SPL on R2R and 3.7% absolute in SR on RxR. Our analysis reveals even larger gains for navigation instructions that contain six or more object references, which further suggests that our approach is better able to use object features and align them to references in the instructions.
Introduction
Vision-and-Language Navigation (VLN) requires an AI agent to follow natural language instructions, such as "Walk past the kitchen and stop by the wooden table," within a 3D environment. Success hinges on the agent's ability to "ground" linguistic tokens to visual landmarks. Current research faces two primary hurdles: the dual nature of landmarks and the domain gap in pretraining.
Instructions typically mention both broad scenes (e.g., "hallway") and specific objects (e.g., "desk"). Most existing models rely on a single visual encoder trained for image classification, which often fails to capture both scales effectively. Furthermore, while multimodal models pretrained on web data excel at object-level features, VLN agents typically use image-level features, creating a "domain shift" that limits the benefits of large-scale pretraining.
The SOAT Architecture
The proposed Scene- and Object-Aware Transformer (SOAT) builds upon the recurrent VLN$\circlearrowright$BERT architecture to explicitly model the world as a collection of high-level scenes and specific objects.
Dual Visual Encoders
SOAT employs two separate pipelines to extract visual cues. It uses a ResNet-152 (trained on the Places dataset) for scene-level features and a Faster R-CNN detector (trained on Visual Genome) for object-level features. This dual-stream approach ensures that the model receives both broad contextual information and fine-grained object details.
Selective Object Attention and View Aggregation
To prevent overfitting when integrating additional features, SOAT introduces a novel Selective Object Attention mechanism. In this setup, scene features and instruction tokens are "frozen," acting only as keys and values. Only the "state token" (the agent's memory) and the object features are allowed to be updated. This forces the model to use the scene as a stable anchor to refine its understanding of objects within that context.
Additionally, SOAT utilizes a View Aggregation Strategy. Since a single viewpoint may contain multiple objects alongside a scene description, the model employs a max-pooling style approach over attention scores. It selects the most relevant feature—whether a specific object or the scene itself—to represent each potential movement direction.
Experimental Evaluation
The model was evaluated on two primary indoor navigation benchmarks:
- Room-to-Room (R2R): The standard benchmark featuring shorter, direct instructions.
- Room-Across-Room (RxR): A more challenging, multilingual dataset with longer instructions and complex paths.
Key Results
SOAT demonstrated significant improvements over previous state-of-the-art models:
- R2R: Improved Success weighted by Path Length (SPL) by 1.8% absolute in unseen environments.
- RxR: Improved Success Rate (SR) by 3.7% and path adherence (NDTW) by 2.4%.
- Object-Heavy Instructions: On RxR instructions with six or more object references, SOAT outperformed the baseline by 7.9% in Success Rate, confirming its superior ability to ground specific object mentions.
Discussion and Future Work
A key insight is that architecture matters as much as data; simply adding object features to a standard model without the selective attention and aggregation strategies actually decreased performance. Furthermore, SOAT's superiority is most evident when initialized with vision-and-language pretraining weights (e.g., OSCAR), suggesting it is better at "unlocking" knowledge from large-scale web data.
Future work aims to address the "viewpoint gap" between well-framed web photos used for training detectors and the panoramic house scans used in VLN. Additionally, the authors plan to extend the model to multilingual instructions and continuous environments where navigation is not restricted to a fixed graph.
Summary updated