Explicit Object Relation Alignment for Vision and Language Navigation

Yue Zhang, Parisa KordjamshidiACL 2022 · OpenReview

Abstract

In this paper, we investigate the problem of vision and language navigation. To solve this problem, grounding the landmarks and spatial relations in the textual instructions into visual modality is important. We propose a neural agent named Explicit Object Relation Alignment Agent (EXOR), to explicitly align the spatial information in both instruction and the visual environment, including landmarks and spatial relationships between the agent and landmarks. Empirically, our proposed method surpasses the baseline by a large margin on the R2R dataset. We provide a comprehensive analysis to show our model’s spatial reasoning ability and explainability.

Introduction

Vision and Language Navigation (VLN) requires an autonomous agent to navigate through photo-realistic 3D environments by following natural language instructions. A fundamental challenge in VLN is grounding: the agent must map linguistic references (e.g., "table") to specific visual features in its environment. While existing models have made strides in object identification, they often struggle with spatial context—distinguishing between a "table on the left" and a "table on the right." Without explicit spatial reasoning, agents frequently fail to generalize to unseen environments or deviate from the intended path.

Methodology: The EXOR Agent

To address these challenges, we propose the Explicit Object Relation Alignment Agent (EXOR). EXOR utilizes a four-stage pipeline designed to bridge the gap between textual instructions and visual perception through explicit alignment.

The architecture of the EXOR model, highlighting the Object Relation Alignment Module.

The pipeline consists of:

  1. Spatial Configuration Representation: Instructions are decomposed into spatial configurations, each containing a motion verb, a landmark, and a spatial indicator.
  2. Landmark Selection: An attention mechanism ranks landmarks, selecting the top-$k$ most relevant ones to minimize visual noise.
  3. Landmark-Object Alignment: A pre-trained Faster-RCNN detects objects in the panoramic view, and cosine similarity is used to match these detections with textual landmarks.
  4. Spatial Relation Alignment: This core innovation encodes spatial relations (left, right, front, back, up, down) into a 6-bit vector. The agent verifies its heading by comparing textual spatial indicators with the relative visual angles of detected objects.

Experimental Setup and Results

We evaluated EXOR on the Room-to-Room (R2R) dataset within the Matterport3D environment. The dataset includes over 21,000 instructions across seen and unseen indoor environments.

EXOR demonstrated significant improvements over the Env_Drop baseline. In unseen environments, EXOR achieved a 52% Success Rate (SR), compared to 47% for the baseline. Furthermore, EXOR showed high path fidelity, as measured by Success weighted by normalized Dynamic Time Warping (SDTW), indicating that the agent accurately follows the prescribed trajectory rather than reaching the goal by chance.

Discussion: Optimal Landmark Grounding

A key finding in our analysis is the "Goldilocks" effect regarding the number of landmarks considered at each step.

Impact of top-k landmark selection on navigation performance (SPL) in seen and unseen environments.

As shown in the figure above, performance peaks when the agent focuses on exactly three landmarks ($k=3$). Selecting too few landmarks provides insufficient context, while selecting too many introduces noise from distant or irrelevant objects. This explicit alignment also enhances explainability, allowing researchers to visualize which objects the agent prioritizes during navigation.

Conclusion

EXOR improves VLN by explicitly aligning landmarks and spatial relations. While the model's performance is currently constrained by the accuracy of pre-trained object detectors, it offers a transparent and geometrically grounded approach to navigation. Future work will explore integrating this explicit alignment logic into large-scale Transformer-based architectures to combine robust spatial reasoning with massive pre-training.

Summary updated

Abstract

In this paper, we investigate the problem of vision and language navigation. To solve this problem, grounding the landmarks and spatial relations in the textual instructions into visual modality is important. We propose a neural agent named Explicit Object Relation Alignment Agent (EXOR),to explicitly align the spatial information in both instruction and the visual environment, including landmarks and spatial relationships between the agent and landmarks. Empirically, our proposed method surpasses the baseline by a large margin on the R2R dataset. We provide a comprehensive analysis to show our model’s spatial reasoning ability and explainability.
BibTeX
@inproceedings{zhang2022explicit,
  author = {Yue Zhang and Parisa Kordjamshidi},
  title = {{Explicit Object Relation Alignment for Vision and Language Navigation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=kzfv8BnCvo},
}
Figure