Narrowing the Gap between Vision and Action in Navigation

Yue Zhang, Parisa Kordjamshidi

Abstract

Existing methods for Vision and Language Navigation in Continuous Environments (VLN-CE) typically utilize waypoint predictors to discretize the environment, simplifying navigation into a view selection task. While this improves performance over direct low-level action training, significant gaps remain between visual perception and executed actions. Specifically, agents often ignore spatial reasoning within low-level movements and neglect object semantics related to passability. This paper introduces a dual-action framework featuring a low-level action decoder jointly trained with high-level action prediction. Additionally, we enhance waypoint prediction by incorporating semantic-rich visual representations and explicit obstacle masking based on human prior knowledge. Empirical results demonstrate that our approach significantly improves navigation performance across both high-level and low-level metrics on standard benchmarks.

Introduction

Vision and Language Navigation (VLN) requires an agent to follow natural language instructions (e.g., "Walk towards the pool table") within photo-realistic environments. While early research focused on discrete environments where agents jumped between pre-defined nodes, the field has shifted toward VLN-CE, which utilizes continuous 3D spaces. In these settings, agents must execute low-level controls such as FORWARD 0.25m or TURN 15°.

To manage the complexity of continuous space, current VLN-CE agents often use a "waypoint predictor" to discretize the environment into reachable points. However, this creates two major gaps:

  1. The Reasoning Gap: Agents trained primarily on high-level view selection often fail to learn the spatial logic required for the low-level movements between those views.
  2. The Semantic Gap: Existing predictors often rely on basic visual features (like RGB/Depth) but ignore object semantics. They do not inherently recognize that a sofa is an obstacle while a floor is a navigable surface.

Methodology

To bridge these gaps, we propose two primary innovations integrated into a dual-action framework.

Obstacle-Aware Waypoint Predictor

Rather than relying on standard ResNet features, we utilize Vision-Language Pre-trained Models (VLPMs) like CLIP, which provide richer semantic information. We implement an Obstacle Mask mechanism to incorporate human-like prior knowledge:

  • Semantic Segmentation: The model identifies specific objects in the scene.
  • Passability Vocabulary: We define a set of "passable" objects (e.g., floor, stairs, door).
  • Explicit Masking: The model masks out "impassable" objects (e.g., tables, sofas, fireplaces), forcing the waypoint predictor to focus exclusively on open, navigable areas.

Dual-Action Module

We equip the navigator with a Transformer-based low-level action decoder. This module enables Joint Training, where the agent is trained to simultaneously select a high-level viewpoint and generate the exact sequence of low-level actions (e.g., left, left, forward) required to reach it. By treating low-level action generation as a sequence generation task—similar to how a language model predicts tokens—we encourage the agent to ground its visual perception directly into physical motion.

Experimental Results

Our study utilizes the R2R-CE (Room-to-Room in Continuous Environments) dataset, based on Matterport3D (MP3D), and evaluated within the Habitat Simulator. We applied our modules to several strong baselines, including WP-VLN-BERT, WP-HAMT, and ETPNav.

Key Findings

  • Performance Gains: On the R2R-CE test unseen set, the WP-HAMT+Ours model improved the Success Rate (SR) from 49% to 52%.
  • Low-Level Execution: The improvements were most pronounced in low-level action execution. For the WP-HAMT baseline, the Success Rate jumped from 35% to 44% on the validation unseen set after adding the dual-action module.
  • Semantic Accuracy: The obstacle mask improved "Open Area" prediction accuracy by 5% compared to baseline models, proving that semantic knowledge of object passability is a significant boost for robotic agents.
  • State-of-the-Art: The combination of ETPNav with our proposed modules achieved a new SOTA Success Rate of 58% on the R2R-CE validation unseen set.

Discussion and Conclusion

Our research demonstrates that spatial grounding and semantic awareness are critical for narrowing the gap between vision and action. Training an agent to "speak" its low-level actions via a Transformer decoder acts as a powerful regularizer, forcing the model to understand the relationship between visual pixels and physical steps.

While the dual-action framework introduces some computational overhead due to the Transformer-based decoder, the gains in navigation success—particularly in unseen environments—justify the cost. Future work will focus on further integrating vision-language foundation models to close the remaining performance gap between high-level planning and low-level execution.

Summary updated

Abstract

The existing methods for Vision and Language Navigation in the Continuous Environment (VLN-CE) commonly incorporate a waypoint predictor to discretize the environment. This simplifies the navigation actions into a view selection task and improves navigation performance significantly compared to direct training using low-level actions. However, the VLN-CE agents are still far from the real robots since there are gaps between their visual perception and executed actions. First, VLN-CE agents that discretize the visual environment are primarily trained with high-level view selection, which causes them to ignore crucial spatial reasoning within the low-level action movements. Second, in these models, the existing waypoint predictors neglect object semantics and their attributes related to passibility, which can be informative in indicating the feasibility of actions. To address these two issues, we introduce a low-level action decoder jointly trained with high-level action prediction, enabling the current VLN agent to learn and ground the selected visual view to the low-level controls. Moreover, we enhance the current waypoint predictor by utilizing visual representations containing rich semantic information and explicitly masking obstacles based on humans' prior knowledge about the feasibility of actions. Empirically, our agent can improve navigation performance metrics compared to the strong baselines on both high-level and low-level actions.
BibTeX
@inproceedings{zhang2024narrowing,
  author = {Yue Zhang and Parisa Kordjamshidi},
  title = {{Narrowing the Gap between Vision and Action in Navigation}},
  booktitle = {ACM Multimedia},
  year = {2024},
  url = {https://openreview.net/forum?id=vscynaeAJn},
}
Figure