SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation

Abhinav Moudgil, Arjun Majumdar, Harsh Agrawal, Stefan Lee, Dhruv Batra

Introduction

Vision-and-Language Navigation (VLN) agents typically rely on whole-scene visual features, even though human instructions heavily reference specific objects (e.g., "stop by the green chair"). This paper introduces the Scene- and Object-Aware Transformer (SOAT), an agent that explicitly combines both scene-level and object-level visual features. By aligning the model's visual processing with how humans actually describe paths, SOAT achieves significant performance leaps, particularly on complex instructions packed with object references.

The Challenge of Grounding Instructions

In VLN, an AI agent is placed in a 3D environment and must follow natural language instructions to reach a goal. To succeed, the agent must "ground" the words in the instruction to the visual world around it.

Human instructions naturally mix scene descriptions ("exit the bedroom") with object references ("stop at the desk with two green chairs").

Human instructions mix scene and object references, requiring models to process both types of visual cues.

However, most existing VLN models process their visual surroundings using a single Convolutional Neural Network (CNN) trained to classify whole scenes. These models struggle to recognize fine-grained object details. Furthermore, recent state-of-the-art VLN models initialize their networks using multimodal transformers (like OSCAR) pretrained on object bounding boxes from the web. Fine-tuning these models for VLN using scene features creates a massive domain shift. Simply throwing object features into existing VLN architectures actually hurts performance. The challenge lies in architecting a model that can digest both high-level scene context and specific object detections simultaneously.

Proposed Method: SOAT

The authors propose SOAT, which builds upon a baseline called VLN$\circlearrowright$BERT but introduces two distinct visual encoders: a CNN trained on the Places dataset for scene features, and a Faster R-CNN trained on Visual Genome for object features.

Comparison of the baseline VLN-BERT architecture and the proposed SOAT approach, highlighting the integration of object features and view aggregation.

To make these two feature types work together effectively, the authors introduce two key architectural changes:

  1. Selective Object Attention: Instead of letting all inputs attend to each other equally, SOAT uses a custom attention mask. The instruction text and the scene features are "frozen" during the transformer layers—they act only as keys and values. Meanwhile, the agent's state token and the object features act as queries, keys, and values. This forces the model to focus its processing power on refining object representations while using the scene features purely as high-level context.
  2. View Aggregation: At each step, the agent looks at several candidate views (navigable directions). Each view contains one scene feature and zero or more object features. SOAT calculates an attention score for all of them and takes the maximum score to represent that specific view. This allows the model to dynamically decide whether a specific object or the overall scene is the most important cue for choosing its next step.

Key Insights and Results

The researchers evaluated SOAT on two major datasets: the standard Room-to-Room (R2R) benchmark and the newer, more complex Room-Across-Room (RxR) dataset, which features longer paths and nearly double the object references per instruction.

  • Naive concatenation fails: Without custom selective attention and view aggregation mechanisms, adding object features to the baseline actually degraded performance.
  • Pretraining alignment is crucial: The architecture bridges the gap between how multimodal transformers are pretrained (on objects) and fine-tuned (on navigation). Without large-scale vision-and-language pretraining, object features provided almost no benefit.
  • Strong Benchmark Gains: SOAT achieved a 1.8% absolute improvement in Success weighted by Path Length (SPL) over the baseline on the R2R unseen validation split. On the more complex RxR dataset, it achieved a 3.7% absolute improvement in Success Rate (SR).
  • Excelling on Object-Heavy Instructions: The model's improvements are directly tied to its ability to see objects. On a subset of RxR instructions containing 6 or more object references, SOAT outperformed the baseline by a massive 7.9% absolute in Success Rate.

Limitations and Future Work

Like much prior work, SOAT operates on a discrete navigation graph where the agent teleports between predefined viewpoints. Adapting the model for continuous, real-world environments remains a goal for future work. Additionally, while SOAT outperforms its direct baseline, exploring synergies with other pretraining strategies (like augmented VLN data) presents a highly promising direction for further boosting navigation performance.

Summary updated

Abstract

Natural language instructions for visual navigation often use scene descriptions (e.g., bedroom) and object references (e.g., green chairs) to provide a breadcrumb trail to a goal location. This work presents a transformer-based vision-and-language navigation (VLN) agent that uses two different visual encoders -- a scene classification network and an object detector -- which produce features that match these two distinct types of visual cues. In our method, scene features contribute high-level contextual information that supports object-level processing. With this design, our model is able to use vision-and-language pretraining (i.e., learning the alignment between images and text from large-scale web data) to substantially improve performance on the Room-to-Room (R2R) and Room-Across-Room (RxR) benchmarks. Specifically, our approach leads to improvements of 1.8% absolute in SPL on R2R and 3.7% absolute in SR on RxR. Our analysis reveals even larger gains for navigation instructions that contain six or more object references, which further suggests that our approach is better able to use object features and align them to references in the instructions.
BibTeX
@inproceedings{moudgil2021soat,
  author = {Abhinav Moudgil and Arjun Majumdar and Harsh Agrawal and Stefan Lee and Dhruv Batra},
  title = {{SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=P5dtd0yyaz},
}
Figure