Narrowing the Gap between Vision and Action in Navigation

Yue Zhang, Parisa Kordjamshidi

TL;DR

Vision-and-Language Navigation in Continuous Environments (VLN-CE) typically relies on waypoint predictors to turn low-level robotic control into high-level viewpoint selection, ignoring low-level motion dynamics and physical obstacle semantics. This paper bridges this vision–action gap by introducing a dual-action framework—jointly predicting high-level viewpoints and autoregressively decoding low-level action sequences—alongside an obstacle-aware waypoint predictor enhanced with CLIP features and semantic obstacle masking.

What Problem Does This Paper Address?

In realistic continuous environments, embodied agents must navigate using low-level motor primitives (e.g., turning $15^\circ$, moving forward $0.25\text{ m}$). Direct end-to-end training on low-level actions struggles because continuous trajectories are long (~56 steps vs. 4–6 in discrete graphs), compounding errors.

To cope, top VLN-CE methods use an offline waypoint predictor to discretize the local continuous space into candidate viewpoints, training the agent solely as a high-level view selector and handing physical execution over to a heuristic controller. However, this introduces two major gaps:

  1. The Vision–Action Gap: Agents trained purely on viewpoint selection bypass the spatial reasoning embedded in physical trajectories, resulting in weak grounding between instructions, observations, and physical motion.
  2. Semantic Obliviousness: Standard waypoint predictors rely on low-level visual features (e.g., ImageNet-pretrained ResNet) and ignore physical affordances (e.g., walls and tables cannot be traversed, whereas open floors and doorways can).

Methods

The authors tackle these issues with two complementary components:

           ┌────────────────────────┐
           │ Panoramic Observations │
           └───────────┬────────────┘
                       │
       ┌───────────────┴───────────────┐
       ▼                               ▼
┌────────────────────────────┐   ┌────────────────────────────┐
│ Obstacle-Aware Waypoint    │   │ Multimodal Navigator       │
│ Predictor                  │   │ (HAMT / ETPNav Backbone)   │
│ - CLIP Vision Encoder      │   └─────────────┬──────────────┘
│ - Semantic Obstacle Mask   │                 │
└──────────────┬─────────────┘                 │
               ▼                               ▼
       Candidate Viewpoints            [CLS] Representation
               │                               │
               └───────────────┬───────────────┘
                               ▼
                    ┌─────────────────────┐
                    │ Dual-Action Module  │
                    ├─────────────────────┤
                    │ • High-Level Choice │
                    │ • Low-Level Decoder │
                    └─────────────────────┘

1. Obstacle-Aware Waypoint Predictor

  • Semantic Visual Encoders: Replaces the standard ResNet-152 visual backbone with CLIP (ViT-B/16) to capture richer open-vocabulary semantics.
  • Prior-Based Obstacle Masking: Utilizes MP3D semantic segmentation with a defined open-area vocabulary (floor, stairs, doors). Areas outside this passability vocabulary are explicitly masked out before predicting the waypoint heatmap.

2. Dual-Action Navigator

Rather than choosing between high-level view selection or low-level classification:

  • Autoregressive Text Decoder: Formulates low-level action generation as sequence generation. Conditioned on the navigator's multimodal [CLS] token and a prompt (low actions:), a Transformer text decoder predicts a sequence of tokenized motor actions (e.g., left, left, forward).
  • Joint Optimization: High-level view selection (trained with Imitation and Reinforcement Learning, $\mathcal{L}_{high}$) and low-level sequence generation (trained via cross-entropy loss, $\mathcal{L}_{low}$) are trained end-to-end: $$\mathcal{L} = \mathcal{L}_{high} + \mathcal{L}_{low}$$

Datasets

  • R2R-CE (Room-to-Room in Continuous Environments): Built on the Habitat Simulator using 3D scans from Matterport3D (MP3D).
  • Splits: 61 training scenes, 11 validation unseen scenes, and 18 test unseen scenes.
  • Supervision: Waypoint connectivity graphs and ground-truth continuous paths mapped from discrete R2R trajectories.

Key Insights and Ideas

  • Low-level supervision acts as a spatial regularizer: Forcing the navigator to explain how to physically reach a selected viewpoint improves the high-level viewpoint selection itself.
  • Sequence generation outperforms per-step classification: Framing motor controls as an autoregressive string sequence resolves the optimization bottleneck of standard per-step low-level classifiers.
  • Bigger vision models aren't always better for affordance: While CLIP significantly improved open-space waypoint prediction, video-pretrained foundation models like InternVideo actually deteriorated waypoint prediction accuracy (dropping %Open from 0.80 to 0.65), showing that general video dynamics do not trivially translate to geometric traversability.

Key Results

  • High-Level Navigation (R2R-CE Test Unseen):
  • When integrated into WP-HAMT, Success Rate (SR) increased from 49% to 52%, and SPL increased from 0.45 to 0.47.
  • On the graph-based ETPNav, SR improved from 55% to 56% (and up to 58% on Validation Unseen).
  • Low-Level Navigation (R2R-CE Validation Unseen):
  • Setting a new state-of-the-art for low-level execution on WP-HAMT, jumping from 35% SR / 0.32 SPL (standard classifier) to 44% SR / 0.38 SPL.
  • On ETPNav + Ours, reached 48% SR and 0.42 SPL, outperforming prior low-level sequence-to-sequence models (e.g., WS-MGMap's 39% SR).
  • Waypoint Predictor Ablation:
  • Combining CLIP features and semantic masking increased the ratio of navigable waypoints placed in open space (%Open) from 0.80 to 0.85, reducing Hausdorff distance ($d_H$) from 2.00 to 1.94.

Limitations and Future Work

  • Computational Overhead: Training the autoregressive Transformer action decoder alongside the cross-modal navigator introduces non-trivial compute costs (training required 6 NVIDIA RTX GPUs for 2.5 days).
  • Residual High-vs-Low Performance Gap: While narrowing the divide, a performance gap remains between executing with high-level teleports/controllers (54% SR on val unseen) versus decoded low-level controls (44% SR).
  • Dataset Artifacts: The R2R-CE continuous trajectories were generated via shortest-turn path heuristics from discrete graphs. Consequently, decoded low-level actions can sometimes reach the intended target while technically disagreeing with linguistic turn instructions.

Summary updated

Abstract

The existing methods for Vision and Language Navigation in the Continuous Environment (VLN-CE) commonly incorporate a waypoint predictor to discretize the environment. This simplifies the navigation actions into a view selection task and improves navigation performance significantly compared to direct training using low-level actions. However, the VLN-CE agents are still far from the real robots since there are gaps between their visual perception and executed actions. First, VLN-CE agents that discretize the visual environment are primarily trained with high-level view selection, which causes them to ignore crucial spatial reasoning within the low-level action movements. Second, in these models, the existing waypoint predictors neglect object semantics and their attributes related to passibility, which can be informative in indicating the feasibility of actions. To address these two issues, we introduce a low-level action decoder jointly trained with high-level action prediction, enabling the current VLN agent to learn and ground the selected visual view to the low-level controls. Moreover, we enhance the current waypoint predictor by utilizing visual representations containing rich semantic information and explicitly masking obstacles based on humans' prior knowledge about the feasibility of actions. Empirically, our agent can improve navigation performance metrics compared to the strong baselines on both high-level and low-level actions.
BibTeX
@inproceedings{zhang2024narrowing,
  author = {Yue Zhang and Parisa Kordjamshidi},
  title = {{Narrowing the Gap between Vision and Action in Navigation}},
  booktitle = {ACM Multimedia},
  year = {2024},
  url = {https://openreview.net/forum?id=vscynaeAJn},
}

Loading related content...

Figure