History Aware Multimodal Transformer for Vision-and-Language Navigation

Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, Ivan Laptev

Abstract

Vision-and-language navigation (VLN) aims to build autonomous visual agents that follow instructions and navigate in real scenes. To remember previously visited locations and actions taken, most approaches to VLN implement memory using recurrent states. Instead, we introduce a History Aware Multimodal Transformer (HAMT) to incorporate a long-horizon history into multimodal decision making. HAMT efficiently encodes all the past panoramic observations via a hierarchical vision transformer (ViT), which first encodes individual images with ViT, then models spatial relation between images in a panoramic observation and finally takes into account temporal relation between panoramas in the history. It, then, jointly combines text, history and current observation to predict the next action. We first train HAMT end-to-end using several proxy tasks including single step action prediction and spatial relation prediction, and then use reinforcement learning to further improve the navigation policy. HAMT achieves new state of the art on a broad range of VLN tasks, including VLN with fine-grained instructions (R2R, RxR), high-level instructions (R2R-Last, REVERIE), dialogs (CVDN) as well as long-horizon VLN (R4R, R2R-Back). We demonstrate HAMT to be particularly effective for navigation tasks with longer trajectories.

Introduction

In Vision-and-Language Navigation (VLN), agents must interpret natural language instructions to navigate through complex 3D environments. A persistent challenge in this field is effective memory management. Traditional agents rely on Recurrent Neural Networks (RNNs) like LSTMs to maintain state. However, these models suffer from a "recurrent bottleneck," where history is compressed into a fixed-size vector, leading to the loss of crucial information over long trajectories.

While Transformers offer a potential solution for modeling long-range dependencies, their application to VLN is computationally expensive due to the high dimensionality of visual inputs (e.g., 360-degree panoramic views). To address this, the History Aware Multimodal Transformer (HAMT) introduces a hierarchical approach to encode long-horizon history efficiently.

The HAMT Architecture

The HAMT architecture consists of unimodal encoders for text, history, and current observations, followed by a cross-modal transformer encoder that predicts the next action.

The overall architecture of the HAMT, showing the flow from unimodal inputs to multimodal action prediction.

The model takes three primary inputs:

  1. Text: Natural language instructions.
  2. History: A sequence of past panoramic observations and actions.
  3. Observation: The current panoramic view of the environment.

Hierarchical History Encoding

The core innovation of HAMT is its Hierarchical Vision Transformer (ViT) for history encoding. Rather than processing every image from every past step as a flat sequence, HAMT uses a three-stage process:

  • View Level: Individual images within a panorama are encoded using a standard ViT.
  • Spatial Level (Panoramic Transformer): This stage models the spatial relationships between different views within a single 360-degree step.
  • Temporal Level (Temporal Transformer): This stage captures the temporal dynamics across the sequence of panoramas in the agent's history.
The Hierarchical Vision Transformer (ViT) encodes history through view-level, spatial-level (panorama), and temporal-level transformers.

This hierarchy allows the model to maintain a detailed memory of long paths while remaining computationally feasible by significantly reducing the number of tokens required for cross-modal attention.

Training Strategy and Proxy Tasks

HAMT is trained in two stages. First, it undergoes multi-task pre-training using several proxy tasks designed to align text and visual history. These include standard tasks like Masked Language Modeling (MLM) and a novel Spatial Relationship Prediction (SPREL) task, which requires the model to predict the relative positions of objects, enhancing its spatial reasoning.

Following pre-training, the model is fine-tuned using Reinforcement Learning (RL) to optimize for navigation success and path fidelity.

Experimental Results

HAMT was evaluated on a comprehensive set of VLN benchmarks, setting new state-of-the-art records across the board:

  • R2R and RxR: Achieved a 65% success rate on the R2R unseen test set, outperforming previous models like RecBERT.
  • Long-Horizon Tasks (R4R and R2R-Back): HAMT demonstrated significant improvements in path fidelity (SDTW). Notably, on the R2R-Back task—which requires returning to the starting point—HAMT achieved a 57.2% success rate, a substantial jump from the 41.1% achieved by recurrent models.
  • High-Level Instructions: Showed strong performance on REVERIE and R2R-Last, proving its ability to generalize from sparse instructions.

Conclusion

The History Aware Multimodal Transformer demonstrates that explicit, hierarchical history encoding is superior to implicit recurrent states for complex navigation tasks. By efficiently managing long-horizon visual memory, HAMT enables agents to follow intricate instructions and navigate successfully over long distances.

Summary updated

Abstract

Vision-and-language navigation (VLN) aims to build autonomous visual agents that follow instructions and navigate in real scenes. To remember previously visited locations and actions taken, most approaches to VLN implement memory using recurrent states. Instead, we introduce a History Aware Multimodal Transformer (HAMT) to incorporate a long-horizon history into multimodal decision making. HAMT efficiently encodes all the past panoramic observations via a hierarchical vision transformer (ViT), which first encodes individual images with ViT, then models spatial relation between images in a panoramic observation and finally takes into account temporal relation between panoramas in the history. It, then, jointly combines text, history and current observation to predict the next action. We first train HAMT end-to-end using several proxy tasks including single step action prediction and spatial relation prediction, and then use reinforcement learning to further improve the navigation policy. HAMT achieves new state of the art on a broad range of VLN tasks, including VLN with fine-grained instructions (R2R, RxR), high-level instructions (R2R-Last, REVERIE), dialogs (CVDN) as well as long-horizon VLN (R4R, R2R-Back). We demonstrate HAMT to be particularly effective for navigation tasks with longer trajectories.
BibTeX
@inproceedings{chen2021history,
  author = {Shizhe Chen and Pierre-Louis Guhur and Cordelia Schmid and Ivan Laptev},
  title = {{History Aware Multimodal Transformer for Vision-and-Language Navigation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=x2rnCcghSkz},
}
Figure