SEDualVLN: A Spatially-Enhanced Dual-System for Vision-Language Navigation

Jingzhi Huang, Junkai Huang, Wenxuan Song, Haoyang Yang, Hailong Huang, Haoang Li, Yi WangarXiv 2026 · arXiv

TL;DR

SEDualVLN introduces a spatially-enhanced, dual-system framework for Vision-Language Navigation in continuous environments (VLN-CE). By pairing a fast, reactive Vision-Language Model (VLM) for low-level action generation with a slow, deliberative Multimodal Large Language Model (MLLM) for high-level waypoint planning, the framework achieves state-of-the-art navigation performance without sacrificing execution efficiency.

What Problem Does This Paper Address?

Vision-Language Navigation (VLN) requires an embodied agent to navigate complex, unseen 3D environments guided by natural language instructions. Current approaches generally fall into two paradigms, each with distinct trade-offs:

  1. End-to-end VLM policies: These models are fast and directly map visual streams to actions, but they struggle with long-horizon tasks and lack the deep, step-wise reasoning needed to recover from mistakes.
  2. Zero-shot modular pipelines: These leverage powerful MLLMs (like GPT-4o) as high-level planners. While they excel at commonsense reasoning, they lack fine-grained spatial and geometric understanding, frequently suffer from spatial hallucinations, and incur massive computational/time delays if queried at every step.

SEDualVLN bridges this gap by mimicking the human brain's dual-process theory: a fast, intuitive "System 1" handles immediate actions, while a slow, logical "System 2" handles long-term planning.

Figure 1: Overview of SEDualVLN. System 1 (orange) generates fast, low-level actions from RGB streams via a VLM, while System 2 (green) plans waypoints using top-down 3D map and rendering path images through a general MLLM. The systems cooperate in a fast-slow coordinated loop.

Methods

The SEDualVLN framework coordinates two distinct systems through a fast-slow loop (typically operating at a 20:1 frequency ratio):

System 1: Spatially-Enhanced VLM (Fast-Loop, ~20Hz)

Built upon the StreamVLN backbone (using LLaVA-Video-7B), System 1 generates low-level atomic actions (e.g., MoveForward, TurnLeft). To prevent metric drift and wrong turns, the authors introduce two spatial enhancements:

  • Global Spatial Enhancement Strategy: The model's intermediate attention layers are implicitly supervised using 3D spatial features from a pretrained 3D foundation model (VGGT). This guides the visual token embeddings to encode environmental geometry without requiring active depth sensors during deployment.
  • Local Spatial Enhancement Strategy: A Channel Connectivity Extraction Module uses Grounding DINO and SAM to detect and segment nearby passages (corridors, doors). Non-passage regions are masked out, forcing the VLM to explicitly focus on topological connectivity.
Figure 2: Framework of System 1. Given input RGB image streams and language instructions, the model first extracts four types of tokens. Tokens derived from VGGT[37] are processed through the Global Spatial Enhancement Strategy, which provides implicit supervision to Vision Token 1 within the VLM’s attention layers. The remaining three token types are passed through Voxel-based Spatial Pruning and Cache before being fed into the VLM model.

System 2: Mapping–Rendering–Reasoning MLLM (Slow-Loop, ~1Hz)

When invoked, System 2 acts as a deliberative waypoint planner through a three-stage pipeline:

  1. Mapping: Incrementally builds a lightweight 3D map and a 2D frontier map online using LingBot-Map.
  2. Rendering: Projects 2D frontier points into the 3D map, plans collision-free paths using A*, and renders first-person virtual camera views along those paths. Redundant frames are pruned using CLIP similarity.
  3. Reasoning: GPT-4o evaluates the top-down 3D map alongside the rendered path image sequences, selecting the optimal frontier waypoint that best matches the language instruction.
Figure 3: System 2 Workflow. It consists of three stages: Mapping, which builds a 3D map and 2D frontier map; Rendering, which generates virtual camera views along planned paths; and Reasoning, where the agent first interprets the environment and then evaluates frontier paths against the instruction to select the optimal navigation target.

Datasets

The framework was evaluated on two standard continuous-environment VLN benchmarks based on Matterport3D reconstructions:

  • R2R-CE (Room-to-Room Continuous Environments): Focuses on short-to-medium range English instructions.
  • RxR-CE (Room-Across-Room Continuous Environments): Features longer, multi-room trajectories with dense multilingual instructions.

Key Insights and Ideas

  • Mid-Layer Supervision is Key: Ablation studies revealed that supervising intermediate layers (specifically layer 24 of the 32-layer VLM) with 3D geometric features is significantly more effective than supervising the deepest layers. Deeper layers tend to lose low-level visual-spatial details in favor of high-level semantics.
  • Passage Masking Prevents Irreversible Errors: Most VLM navigation failures occur when an agent blindly enters an incorrect room or hallway. Explicitly isolating passage connectivity via open-vocabulary segmentation drastically reduces junction errors.
  • The Power of Coordinated Frequencies: Running the MLLM planner at every step is prohibitively slow. By allowing the VLM to execute multiple low-level actions while the MLLM computes the next major waypoint, the agent minimizes idle reasoning delays while maintaining global awareness.

Key Results

SEDualVLN achieved new state-of-the-art performance on the competitive Val-Unseen splits for both benchmarks, outperforming previous dual-system and end-to-end models:

  • R2R-CE: Achieved a 67.3% Success Rate (SR) and 62.5% Success weighted by Path Length (SPL), outperforming the prior best dual-system method (DualVLN) by 3.0% SR.
  • RxR-CE: Achieved a 63.9% SR and 52.4% SPL, a 2.5% SR improvement over DualVLN.

From an efficiency standpoint, collaborating System 1 and System 2 at a 20:1 frequency ratio reduced the average episode completion time by a factor of six compared to running the MLLM single-system alone (57.3 seconds vs. 294.3 seconds), while achieving higher overall success.

Limitations and Future Work

The primary bottleneck identified by the authors is deployment on physical robotic hardware. Current real-time 3D mapping techniques still face performance constraints in complex real-world environments. This can lead to distorted 3D map reconstructions, which subsequently degrades the rendering and reasoning accuracy of the System 2 planner. Future work will likely target more robust, noise-tolerant online 3D mapping modules.

Summary updated

Abstract

Vision-Language Navigation (VLN) approaches have currently followed two primary paradigms: the end-to-end Vision-Language Model (VLM) policy fine-tuned on navigation trajectories to directly predict actions, and the zero-shot modular pipeline integrating pre-trained Multimodal Large Language Model (MLLM) for training-free generalization to unseen environments. However, end-to-end methods struggle with long-horizon navigation and lack dynamic reasoning, whereas zero-shot methods are constrained by limited spatial grounding for reliable planning and also require substantial reasoning time. To bridge this gap, we introduce SEDualVLN, a spatially-enhanced dual-system VLN framework. System 1 is a VLM model enhanced with both global and local spatial awareness, used for action generation. System 2 integrates a general MLLM with a mapping module, wherein the MLLM plans waypoints by leveraging top-down views of the real-time 3D map alongside streams of rendered path images. Both systems leverage different forms of spatial enhancement to cultivate the agent's sense of direction in VLN tasks. Ultimately, they cooperate to complete the navigation task through a fast-slow coordinated approach. SEDualVLN achieves state-of-the-art performance on VLN-CE benchmarks, and further ablation studies demonstrate the effectiveness of each system and module.
BibTeX
@misc{huang2026sedualvln,
  author = {Jingzhi Huang and Junkai Huang and Wenxuan Song and Haoyang Yang and Hailong Huang and Haoang Li and Yi Wang},
  title = {{SEDualVLN: A Spatially-Enhanced Dual-System for Vision-Language Navigation}},
  year = {2026},
  eprint = {2605.17249},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.17249},
}

Loading related content...

Figure