Target-Driven Structured Transformer Planner for Vision-Language Navigation

Yusheng Zhao, Jinyu Chen, Chen Gao, Wenguan Wang, Lirong Yang, Haibing Ren, Huaxia Xia, Si Liu

Introduction

Vision-Language Navigation (VLN) is a critical task in embodied AI, requiring agents to navigate complex 3D environments based on natural language instructions. While significant progress has been made, many current agents suffer from "myopia"—a tendency to focus excessively on immediate visual surroundings and past trajectories while lacking a coherent sense of the long-term goal. Furthermore, standard models often treat navigation history as a flat sequence of observations, ignoring the inherent topological structure of physical environments.

To address these challenges, we present the Target-Driven Structured Transformer Planner (TD-STP). This framework introduces a target-estimation mechanism that allows agents to "imagine" their final destination and a structured transformer that respects the physical layout of the environment, leading to more efficient and successful navigation.

Methodology

TD-STP introduces three primary innovations to enhance spatial reasoning and long-term planning:

1. Imaginary Scene Tokenization (IST)

To mitigate the issue of myopic decision-making, the IST mechanism provides the agent with a "North Star." The unexplored environment is discretized into a fixed-size grid of "target tokens." At the beginning of a task, the agent uses the linguistic instructions to make an initial prediction of which grid cell contains the final destination. As the agent moves and gathers new visual data, it progressively refines this "imagined" target. This continuous refinement allows the agent to maintain a spatial anchor even when the goal is not yet visible.

2. Structured Transformer Planner (STP)

Standard Transformers often struggle to capture the connectivity of a physical space. The STP addresses this by building a graph representation of the environment. The adjacency matrix of this graph is used to mask the Transformer’s attention mechanism, ensuring that information flow is restricted to locations that are physically connected. This topological awareness helps the agent learn a more robust representation of the house layout and improves generalization to unseen environments.

3. Global Action Space

Unlike traditional agents limited to local movements, TD-STP operates within a global action space. The agent can choose to "jump" back to any previously visited location in its navigation graph. This capability is essential for efficient backtracking; if the agent’s refined "imagined" target suggests it has taken a wrong turn, it can immediately return to a previous junction rather than wandering aimlessly.

Experimental Results

The TD-STP framework was evaluated on two primary VLN benchmarks: Room-to-Room (R2R), which focuses on fine-grained instructions, and REVERIE, which requires finding specific remote objects based on high-level descriptions.

Performance on R2R

On the R2R "Test Unseen" split, TD-STP achieved a 67% Success Rate (SR). This represents a 2% absolute improvement over previous state-of-the-art (SOTA) models like HAMT. The model also demonstrated higher Success weighted by Path Length (SPL), indicating more direct and efficient navigation paths.

Performance on REVERIE

The improvements were even more pronounced on the REVERIE benchmark. TD-STP achieved a 35.89% Success Rate on the "Test Unseen" split, a significant 5.49% jump over the previous SOTA. This suggests that the target-driven approach is particularly effective for high-level tasks where the destination is not explicitly described in step-by-step detail.

Discussion and Future Work

The success of TD-STP highlights two key insights: the importance of spatial "imagination" in long-term planning and the necessity of respecting environmental topology in transformer architectures. By anchoring navigation to a predicted target and constraining attention to physical connections, the agent achieves a more human-like understanding of navigation.

A current limitation of the model is its reliance on a pre-defined environment graph. In real-world scenarios, robots must operate in completely unknown spaces without prior maps. Future research will focus on integrating Simultaneous Localization and Mapping (SLAM) techniques, allowing the agent to construct its own structured map and environment graph dynamically while simultaneously refining its predicted destination.

Summary updated

BibTeX
@inproceedings{zhao2022target,
  author = {Yusheng Zhao and Jinyu Chen and Chen Gao and Wenguan Wang and Lirong Yang and Haibing Ren and Huaxia Xia and Si Liu},
  title = {{Target-Driven Structured Transformer Planner for Vision-Language Navigation}},
  booktitle = {ACM Multimedia},
  year = {2022},
  url = {https://openreview.net/forum?id=gMMFz3rxezX},
}
Figure