Enhancing scene understanding and modal alignment in vision-language navigation via spatial–temporal graph

Hui Li, Yu WangMeasurement 2025 · DOI

Abstract

Abstract Planning paths according to instructions in a complex environment is the current great challenge for the vision language navigation (VLN) task. Most research focuses on capturing individual visual representations of the objects in the current scene to identify the path matching the instruction. However, these methods cannot capture the correlation between objects within a scene as well as between consecutive scenes, resulting in incomprehensive scene and temporal understanding. To address this challenge, we propose a spatial–temporal-enhanced VLN model (STE-VLN). In this model, the spatial–temporal graph is proposed to enhance the model’s understanding of historical and current scenes by integrating spatial positional relationships of objects at adjacent moments. Meanwhile, a graph-wise attention module is proposed to screen visual graph information aligned with instructions and make more appropriate path planning. When evaluated in the target-oriented VLN dataset REVERIE and the room-oriented dataset R2R-LAST, STE-VLN significantly outperforms current state-of-the-art methods across multiple metrics like success rate and remote grounding success. Experimental results show our STE-VLN outperforms all metrics on a challenging coarse-grained VLN R2R-LAST and REVERIE dataset, exceeding the baseline by 3% and 2.92% in success rate respectively. Furthermore, extensive ablation studies and comprehensive analyses have been adopted, proving that the explicit enhancement of the spatial–temporal map and graphical attention improves the model’s path-planning ability and generalization.
BibTeX
@inproceedings{li2025enhancing,
  author = {Hui Li and Yu Wang},
  title = {{Enhancing scene understanding and modal alignment in vision-language navigation via spatial–temporal graph}},
  booktitle = {Measurement},
  year = {2025},
  doi = {10.1088/1361-6501/ae2532},
}

Loading related content...

Figure