Abstract
In this paper, we investigate the problem of vision and language navigation. To solve this problem, grounding the landmarks and spatial relations in the textual instructions into visual modality is important. We propose a neural agent named Explicit Object Relation Alignment Agent (EXOR), to explicitly align the spatial information in both instruction and the visual environment, including landmarks and spatial relationships between the agent and landmarks. Empirically, our proposed method surpasses the baseline by a large margin on the R2R dataset. We provide a comprehensive analysis to show our model’s spatial reasoning ability and explainability.
Introduction
Vision and Language Navigation (VLN) requires an autonomous agent to navigate through photo-realistic 3D environments by following natural language instructions. A fundamental challenge in VLN is grounding: the agent must map linguistic references (e.g., "table") to specific visual features in its environment. While existing models have made strides in object identification, they often struggle with spatial context—distinguishing between a "table on the left" and a "table on the right." Without explicit spatial reasoning, agents frequently fail to generalize to unseen environments or deviate from the intended path.
Methodology: The EXOR Agent
To address these challenges, we propose the Explicit Object Relation Alignment Agent (EXOR). EXOR utilizes a four-stage pipeline designed to bridge the gap between textual instructions and visual perception through explicit alignment.
The pipeline consists of:
- Spatial Configuration Representation: Instructions are decomposed into spatial configurations, each containing a motion verb, a landmark, and a spatial indicator.
- Landmark Selection: An attention mechanism ranks landmarks, selecting the top-$k$ most relevant ones to minimize visual noise.
- Landmark-Object Alignment: A pre-trained Faster-RCNN detects objects in the panoramic view, and cosine similarity is used to match these detections with textual landmarks.
- Spatial Relation Alignment: This core innovation encodes spatial relations (left, right, front, back, up, down) into a 6-bit vector. The agent verifies its heading by comparing textual spatial indicators with the relative visual angles of detected objects.
Experimental Setup and Results
We evaluated EXOR on the Room-to-Room (R2R) dataset within the Matterport3D environment. The dataset includes over 21,000 instructions across seen and unseen indoor environments.
EXOR demonstrated significant improvements over the Env_Drop baseline. In unseen environments, EXOR achieved a 52% Success Rate (SR), compared to 47% for the baseline. Furthermore, EXOR showed high path fidelity, as measured by Success weighted by normalized Dynamic Time Warping (SDTW), indicating that the agent accurately follows the prescribed trajectory rather than reaching the goal by chance.
Discussion: Optimal Landmark Grounding
A key finding in our analysis is the "Goldilocks" effect regarding the number of landmarks considered at each step.
As shown in the figure above, performance peaks when the agent focuses on exactly three landmarks ($k=3$). Selecting too few landmarks provides insufficient context, while selecting too many introduces noise from distant or irrelevant objects. This explicit alignment also enhances explainability, allowing researchers to visualize which objects the agent prioritizes during navigation.
Conclusion
EXOR improves VLN by explicitly aligning landmarks and spatial relations. While the model's performance is currently constrained by the accuracy of pre-trained object detectors, it offers a transparent and geometrically grounded approach to navigation. Future work will explore integrating this explicit alignment logic into large-scale Transformer-based architectures to combine robust spatial reasoning with massive pre-training.
Summary updated