Abstract: Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires agents to navigate unknown, continuous spaces based on natural language instructions. Compared to discrete settings, VLN-CE poses two core perception challenges. First, the absence of predefined observation points leads to heterogeneous visual memories and weakened global spatial correlations. Second, cumulative reconstruction errors in three-dimensional scenes introduce structural noise, impairing local feature perception. To address these challenges, this paper proposes ST-Booster, an iterative spatiotemporal booster that enhances navigation performance through multi-granularity perception and instruction-aware reasoning. ST-Booster consists of three key modules -- Hierarchical SpatioTemporal Encoding (HSTE), Multi-Granularity Aligned Fusion (MGAF), and ValueGuided Waypoint Generation (VGWG). HSTE encodes long-term global memory using topological graphs and captures shortterm local details via grid maps. MGAF aligns these dualmap representations with instructions through geometry-aware knowledge fusion. The resulting representations are iteratively refined through pretraining tasks. During reasoning, VGWG generates Guided Attention Heatmaps (GAHs) to explicitly model environment-instruction relevance and optimize waypoint selection. Extensive comparative experiments and performance analyses are conducted, demonstrating that ST-Booster outperforms existing state-of-the-art methods, particularly in complex, disturbance-prone environments.
Introduction
Vision-and-Language Navigation in Continuous Environments (VLN-CE) represents a significant advancement in complexity over traditional discrete VLN tasks. In discrete settings, agents move between predefined "teleportation" nodes. In contrast, continuous environments require agents to navigate 3D space using low-level controls (e.g., "move forward 0.25m"), mirroring real-world robotic movement. This transition introduces two major challenges: Spatial Disconnect, where the lack of a predefined graph makes it difficult to maintain consistent global memory while attending to local obstacles, and Structural Noise, where 3D reconstructions contain artifacts that degrade local perception.
Existing models typically rely on either topological maps (strong for long-term memory but weak on detail) or grid maps (detailed but computationally expensive for large areas). ST-Booster bridges this gap by iteratively fusing these multi-granularity representations to achieve robust navigation.
Methods
The ST-Booster framework integrates three core modules designed to enhance spatiotemporal perception through iterative interaction:
- Hierarchical SpatioTemporal Encoding (HSTE): This module maintains a dual-map representation. A Topological Map captures long-term global context by recording visited locations as nodes, while a Grid Map (Bird's-Eye-View) provides high-resolution local spatial details.
- Multi-Granularity Aligned Fusion (MGAF): This module facilitates bidirectional knowledge sharing. Cell2Node transfers fine-grained grid details into topological nodes, and Node2Cell projects global historical context back onto the local grid. These representations are aligned with text instructions via cross-modal transformers.
- Value-Guided Waypoint Generation (VGWG): To optimize navigation, VGWG generates Guided Attention Heatmaps (GAHs). These heatmaps explicitly model the relevance between environment features and instructions, highlighting critical landmarks (e.g., "stairs" or "doorway") to guide waypoint selection even in noisy environments.
The model is refined through three pretraining tasks: Masked Language Modeling (MLM), Hybrid Single Action Prediction (HSAP), and Guided Attention Heatmap Prediction (GAHP).
Key Results and Insights
Evaluated on the Matterport3D (MP3D) dataset within the Habitat simulator, ST-Booster established new state-of-the-art performance benchmarks for VLN-CE. It achieved a 61% Success Rate (SR) on the "Val Unseen" split, a 2% improvement over previous leaders. Furthermore, it reached a 50% SPL (Success weighted by Path Length) on the test-unseen split, demonstrating efficient pathfinding.
Key insights from the study highlight the importance of bidirectional fusion. By projecting global history onto the local grid, the agent gains a contextual understanding of its immediate surroundings relative to the overall mission. Additionally, GAHs serve as effective visual filters, steering the agent toward landmarks mentioned in instructions and providing resilience against reconstruction noise.
Conclusion and Future Work
ST-Booster demonstrates that iterative spatiotemporal boosting significantly enhances navigation in continuous 3D spaces. While the model shows remarkable robustness, future challenges include extending perception to "open-world" scenarios with novel objects and managing topological graphs over extremely long durations. Future research will focus on lifelong map maintenance and transitioning from high-fidelity simulation to physical robotic hardware in unpredictable real-world environments.
Summary updated