Introduction
Vision-and-Language Navigation (VLN) agents typically rely on whole-scene visual features, even though human instructions heavily reference specific objects (e.g., "stop by the green chair"). This paper introduces the Scene- and Object-Aware Transformer (SOAT), an agent that explicitly combines both scene-level and object-level visual features. By aligning the model's visual processing with how humans actually describe paths, SOAT achieves significant performance leaps, particularly on complex instructions packed with object references.
The Challenge of Grounding Instructions
In VLN, an AI agent is placed in a 3D environment and must follow natural language instructions to reach a goal. To succeed, the agent must "ground" the words in the instruction to the visual world around it.
Human instructions naturally mix scene descriptions ("exit the bedroom") with object references ("stop at the desk with two green chairs").
However, most existing VLN models process their visual surroundings using a single Convolutional Neural Network (CNN) trained to classify whole scenes. These models struggle to recognize fine-grained object details. Furthermore, recent state-of-the-art VLN models initialize their networks using multimodal transformers (like OSCAR) pretrained on object bounding boxes from the web. Fine-tuning these models for VLN using scene features creates a massive domain shift. Simply throwing object features into existing VLN architectures actually hurts performance. The challenge lies in architecting a model that can digest both high-level scene context and specific object detections simultaneously.
Proposed Method: SOAT
The authors propose SOAT, which builds upon a baseline called VLN$\circlearrowright$BERT but introduces two distinct visual encoders: a CNN trained on the Places dataset for scene features, and a Faster R-CNN trained on Visual Genome for object features.
To make these two feature types work together effectively, the authors introduce two key architectural changes:
- Selective Object Attention: Instead of letting all inputs attend to each other equally, SOAT uses a custom attention mask. The instruction text and the scene features are "frozen" during the transformer layers—they act only as keys and values. Meanwhile, the agent's state token and the object features act as queries, keys, and values. This forces the model to focus its processing power on refining object representations while using the scene features purely as high-level context.
- View Aggregation: At each step, the agent looks at several candidate views (navigable directions). Each view contains one scene feature and zero or more object features. SOAT calculates an attention score for all of them and takes the maximum score to represent that specific view. This allows the model to dynamically decide whether a specific object or the overall scene is the most important cue for choosing its next step.
Key Insights and Results
The researchers evaluated SOAT on two major datasets: the standard Room-to-Room (R2R) benchmark and the newer, more complex Room-Across-Room (RxR) dataset, which features longer paths and nearly double the object references per instruction.
- Naive concatenation fails: Without custom selective attention and view aggregation mechanisms, adding object features to the baseline actually degraded performance.
- Pretraining alignment is crucial: The architecture bridges the gap between how multimodal transformers are pretrained (on objects) and fine-tuned (on navigation). Without large-scale vision-and-language pretraining, object features provided almost no benefit.
- Strong Benchmark Gains: SOAT achieved a 1.8% absolute improvement in Success weighted by Path Length (SPL) over the baseline on the R2R unseen validation split. On the more complex RxR dataset, it achieved a 3.7% absolute improvement in Success Rate (SR).
- Excelling on Object-Heavy Instructions: The model's improvements are directly tied to its ability to see objects. On a subset of RxR instructions containing 6 or more object references, SOAT outperformed the baseline by a massive 7.9% absolute in Success Rate.
Limitations and Future Work
Like much prior work, SOAT operates on a discrete navigation graph where the agent teleports between predefined viewpoints. Adapting the model for continuous, real-world environments remains a goal for future work. Additionally, while SOAT outperforms its direct baseline, exploring synergies with other pretraining strategies (like augmented VLN data) presents a highly promising direction for further boosting navigation performance.
Summary updated