SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation

Abhinav Moudgil, Arjun Majumdar, Harsh Agrawal, Stefan Lee, Dhruv Batra

Abstract

Natural language instructions for visual navigation often use scene descriptions (e.g., "bedroom") and object references (e.g., "green chairs") to provide a breadcrumb trail to a goal location. This work presents a transformer-based vision-and-language navigation (VLN) agent that uses two different visual encoders -- a scene classification network and an object detector -- which produce features that match these two distinct types of visual cues. In our method, scene features contribute high-level contextual information that supports object-level processing. With this design, our model is able to use vision-and-language pretraining to substantially improve performance on the Room-to-Room (R2R) and Room-Across-Room (RxR) benchmarks. Specifically, our approach leads to improvements of 1.8% absolute in SPL on R2R and 3.7% absolute in SR on RxR. Our analysis reveals even larger gains for navigation instructions that contain six or more object references, which further suggests that our approach is better able to use object features and align them to references in the instructions.

Introduction

Vision-and-Language Navigation (VLN) requires an AI agent to follow natural language instructions, such as "Walk past the kitchen and stop by the wooden table," within a 3D environment. Success hinges on the agent's ability to "ground" linguistic tokens to visual landmarks. Current research faces two primary hurdles: the dual nature of landmarks and the domain gap in pretraining.

Instructions typically mention both broad scenes (e.g., "hallway") and specific objects (e.g., "desk"). Most existing models rely on a single visual encoder trained for image classification, which often fails to capture both scales effectively. Furthermore, while multimodal models pretrained on web data excel at object-level features, VLN agents typically use image-level features, creating a "domain shift" that limits the benefits of large-scale pretraining.

Visual cues in VLN: scene descriptions vs. object references.

The SOAT Architecture

The proposed Scene- and Object-Aware Transformer (SOAT) builds upon the recurrent VLN$\circlearrowright$BERT architecture to explicitly model the world as a collection of high-level scenes and specific objects.

Dual Visual Encoders

SOAT employs two separate pipelines to extract visual cues. It uses a ResNet-152 (trained on the Places dataset) for scene-level features and a Faster R-CNN detector (trained on Visual Genome) for object-level features. This dual-stream approach ensures that the model receives both broad contextual information and fine-grained object details.

Comparison between the baseline VLN-BERT (a) and the proposed SOAT approach (b), highlighting the inclusion of object features and the view aggregation strategy.

Selective Object Attention and View Aggregation

To prevent overfitting when integrating additional features, SOAT introduces a novel Selective Object Attention mechanism. In this setup, scene features and instruction tokens are "frozen," acting only as keys and values. Only the "state token" (the agent's memory) and the object features are allowed to be updated. This forces the model to use the scene as a stable anchor to refine its understanding of objects within that context.

Additionally, SOAT utilizes a View Aggregation Strategy. Since a single viewpoint may contain multiple objects alongside a scene description, the model employs a max-pooling style approach over attention scores. It selects the most relevant feature—whether a specific object or the scene itself—to represent each potential movement direction.

Experimental Evaluation

The model was evaluated on two primary indoor navigation benchmarks:

  • Room-to-Room (R2R): The standard benchmark featuring shorter, direct instructions.
  • Room-Across-Room (RxR): A more challenging, multilingual dataset with longer instructions and complex paths.

Key Results

SOAT demonstrated significant improvements over previous state-of-the-art models:

  • R2R: Improved Success weighted by Path Length (SPL) by 1.8% absolute in unseen environments.
  • RxR: Improved Success Rate (SR) by 3.7% and path adherence (NDTW) by 2.4%.
  • Object-Heavy Instructions: On RxR instructions with six or more object references, SOAT outperformed the baseline by 7.9% in Success Rate, confirming its superior ability to ground specific object mentions.

Discussion and Future Work

A key insight is that architecture matters as much as data; simply adding object features to a standard model without the selective attention and aggregation strategies actually decreased performance. Furthermore, SOAT's superiority is most evident when initialized with vision-and-language pretraining weights (e.g., OSCAR), suggesting it is better at "unlocking" knowledge from large-scale web data.

Future work aims to address the "viewpoint gap" between well-framed web photos used for training detectors and the panoramic house scans used in VLN. Additionally, the authors plan to extend the model to multilingual instructions and continuous environments where navigation is not restricted to a fixed graph.

Summary updated

Abstract

Natural language instructions for visual navigation often use scene descriptions (e.g., "bedroom") and object references (e.g., "green chairs") to provide a breadcrumb trail to a goal location. This work presents a transformer-based vision-and-language navigation (VLN) agent that uses two different visual encoders -- a scene classification network and an object detector -- which produce features that match these two distinct types of visual cues. In our method, scene features contribute high-level contextual information that supports object-level processing. With this design, our model is able to use vision-and-language pretraining (i.e., learning the alignment between images and text from large-scale web data) to substantially improve performance on the Room-to-Room (R2R) and Room-Across-Room (RxR) benchmarks. Specifically, our approach leads to improvements of 1.8% absolute in SPL on R2R and 3.7% absolute in SR on RxR. Our analysis reveals even larger gains for navigation instructions that contain six or more object references, which further suggests that our approach is better able to use object features and align them to references in the instructions.
BibTeX
@inproceedings{moudgil2021soat,
  author = {Abhinav Moudgil and Arjun Majumdar and Harsh Agrawal and Stefan Lee and Dhruv Batra},
  title = {{SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=P5dtd0yyaz},
}

Loading related content...

Figure