TL;DR
The History Aware Multimodal Transformer (HAMT) replaces traditional recurrent memory in vision-and-language navigation with an explicit, hierarchical transformer-based encoding of long-horizon history. By jointly attending to instructions, full panoramic history, and current observations, HAMT sets new state-of-the-art benchmarks across multiple navigation tasks, proving especially potent for long-trajectory scenarios where recurrent states typically fail.
What Problem Does This Paper Address?
Vision-and-Language Navigation (VLN) requires an autonomous agent to follow natural language instructions (e.g., "walk past the kitchen, turn left at the stairs, and stop by the double doors") within a 3D environment. A fundamental challenge is memory: the agent must remember where it has been and what it has seen to ground instructions like "return to the start" or "past the sitting area you saw earlier."
Most existing VLN agents use Recurrent Neural Networks (RNNs) to compress history into a fixed-size hidden state. However, these condensed vectors often suffer from "information decay," where early observations fade away, making long-range navigation difficult. While some methods use external maps, they still rely on RNNs for state tracking. HAMT addresses this by explicitly encoding the entire history sequence using a transformer, allowing the agent to attend to any past moment with equal clarity.
Methods
The core of the paper is the HAMT architecture, which consists of three unimodal encoders and a cross-modal fusion module.
- Hierarchical History Encoding: To avoid the quadratic computational cost of attending to every single image in every past panorama, the authors propose a hierarchy:
- View Level: Individual images are encoded using a Vision Transformer (ViT).
- Spatial Level: A "Panoramic Transformer" models relationships between views within a single location.
-
Temporal Level: A "Temporal Transformer" captures the dynamics across different time steps in the trajectory.
This approach, illustrated in Figure 2, balances information integrity with computational efficiency.
- Cross-modal Transformer: This module jointly encodes the text instruction, the hierarchical history, and the current observation to predict the next action. The architecture is shown in Figure 1.
- Two-Stage Training:
- Proxy Tasks: The model is first trained end-to-end on several auxiliary tasks to learn multimodal alignments. These include standard tasks like Masked Language Modeling (MLM) and novel ones like Spatial Relationship Prediction (SPREL) and Single-step Action Prediction (SAP).
- Reinforcement Learning (RL): After proxy task training, the model is fine-tuned using a mix of RL (A3C algorithm) and Imitation Learning (IL) to optimize the navigation policy.
Datasets
The authors evaluate HAMT on a diverse set of benchmarks:
- Standard Benchmarks: R2R (fine-grained instructions), RxR (multilingual), CVDN (dialog-based), and REVERIE (high-level, object-centric).
- Long-Horizon Benchmarks: R4R (concatenated paths).
- New Benchmarks Introduced:
- R2R-Back: Requires the agent to reach a target and then navigate back to the start, testing long-term memory.
- R2R-Last: Uses only the final sentence of an R2R instruction, testing high-level goal seeking without step-by-step guidance.
Key Insights and Ideas
- Explicit History is Superior: Moving from a compressed recurrent state to an explicit sequence allows the model to "look back" at specific landmarks mentioned in instructions that might have been forgotten by an LSTM.
- Efficiency through Hierarchy: By factorizing spatial and temporal attention, HAMT reduces the history encoding complexity from $O(t^2K^2)$ to $O(tK^2 + t^2)$, where $t$ is time and $K$ is the number of views per panorama.
- End-to-End Visual Learning: The paper demonstrates that fine-tuning the visual encoder (ViT) specifically for the navigation task—rather than using fixed features—provides a significant performance boost.
- Proxy Tasks as a Foundation: Large transformers are notoriously hard to train with RL from scratch. The suite of proxy tasks provides a "warm start" that stabilizes subsequent RL fine-tuning.
Key Results
HAMT achieved state-of-the-art (SOTA) results across all tested datasets:
- R2R: Achieved a 66% Success Rate (SR) on the val unseen split, a significant jump over the previous best (RecBERT at 63%).
- Long-Horizon Performance: On the R4R dataset, HAMT improved the normalized Dynamic Time Warping (nDTW) score—a measure of path fidelity—by 9.5% relative to RecBERT.
- Memory Test (R2R-Back): HAMT reached a 57.2% SR on unseen environments, nearly doubling the performance of recurrent models like EnvDrop (32.4%), proving the effectiveness of the explicit history buffer.
Limitations and Future Work
- Object Grounding: Qualitative failure cases show that while the agent is good at navigating, it still occasionally fails to recognize specific objects (e.g., a "liquor shelf") required for the final stop.
- Continuous Environments: The current model operates on a discrete navigation graph. The authors suggest extending HAMT to continuous action spaces as a promising future direction.
- Data Scaling: The model's performance suggests it could benefit from even larger-scale pre-training on diverse navigation data.
Summary updated