Abstract
Deep Learning has revolutionized our ability to solve complex problems such as Vision-and-Language Navigation (VLN). This task requires the agent to navigate to a goal purely based on visual sensory inputs given natural language instructions. However, prior works formulate the problem as a navigation graph with a discrete action space. In this work, we lift the agent off the navigation graph and propose a more complex VLN setting in continuous 3D reconstructed environments. Our proposed setting, Robo-VLN, more closely mimics the challenges of real world navigation. Robo-VLN tasks have longer trajectory lengths, continuous action spaces, and challenges such as obstacles. We provide a suite of baselines inspired by state-of-the-art works in discrete VLN and show that they are less effective at this task. We further propose that decomposing the task into specialized high- and low-level policies can more effectively tackle this task. With extensive experiments, we show that by using layered decision making, modularized training, and decoupling reasoning and imitation, our proposed Hierarchical Cross-Modal (HCM) agent outperforms existing baselines in all key metrics and sets a new benchmark for Robo-VLN.
Introduction
Most Vision-and-Language Navigation (VLN) research operates on "navigation graphs," where agents "teleport" between discrete nodes. While useful for high-level reasoning, this does not reflect the reality of robotics, where a robot must navigate continuous space, avoid physical obstacles, and manage its own linear and angular velocities. Models that perform well on discrete graphs often fail when "lifted" into continuous environments.
To bridge this gap, we introduce Robo-VLN, a benchmark that moves VLN away from simplified discrete graphs and into continuous 3D environments. Robo-VLN requires agents to follow natural language instructions using continuous control and first-person RGB-D observations. The Hierarchical Cross-Modal (HCM) agent achieves a 13% absolute improvement in success rate over standard "flat" models in unseen environments by decoupling high-level linguistic reasoning from low-level motor control.
The Robo-VLN Benchmark
Built upon the Matterport3D dataset and the Habitat simulator, Robo-VLN ports human-annotated instructions from the Room2Room (R2R) dataset into a continuous control formulation. It includes 3,177 trajectories and 9,533 instruction-pairings across 90 environments. Unlike R2R, which has an average path length of 5 steps, Robo-VLN trajectories average 326 steps, making the "long-horizon" problem much more acute.
Hierarchical Cross-Modal (HCM) Agent
The core contribution is the Hierarchical Cross-Modal (HCM) Agent, which splits the navigation task into two specialized layers to decouple reasoning from imitation.
- High-Level Reasoning Module ($\pi_\theta^h$): This module focuses on cross-modal grounding. It uses a Cross-Modal Transformer to align BERT-encoded text instructions with spatial features from RGB and Depth sensors. It maintains a temporal memory via an LSTM and outputs a high-level sub-goal ($a_t^s$), such as a specific displacement or rotation.
- Low-Level Imitator Module ($\pi_\phi^l$): This module focuses on execution. It takes the sub-goal from the high-level module and the current visual state to output continuous velocity commands $(v_t, \omega_t)$. It is trained via imitation learning to mimic an expert controller.
By decoupling these functions, the agent can learn to handle complex instructions at one level and local obstacle avoidance and robot dynamics at another. The training is modularized, allowing both policies to be trained jointly with specialized losses.
Results and Discussion
The HCM agent was compared against several strong "flat" baselines (Seq2Seq, Progress Monitor, and CMA). HCM achieved a 46% success rate (SR) on unseen validation environments, a 13% absolute improvement over the best baseline (33%). In terms of Success weighted by Path Length (SPL), HCM reached 0.40, outperforming the best baseline (0.30) by 10%.
Qualitatively, flat models frequently collided with obstacles or got stuck, while the HCM agent successfully navigated around furniture, demonstrating superior spatial awareness. Key insights include:
- Hierarchy as a Generalization Engine: Decoupling intent from action prevents over-fitting to specific low-level trajectories.
- Separate Modality Alignment: Aligning RGB and Depth features with language separately outperforms early fusion, suggesting instructions refer to distinct geometric or appearance features.
- Continuous Control Complexity: Standard VLN baselines see significant performance drops in Robo-VLN, proving that graph-based assumptions hide real-world challenges.
Conclusion
While Robo-VLN and the HCM agent represent a step toward real-world robotics, limitations remain. The low-level imitator currently relies on expert demonstrations; future work could explore Reinforcement Learning for error recovery. Additionally, moving from simulation to real-world environments will require addressing sensor noise, lighting variations, and dynamic obstacles.
Summary updated