TL;DR
Vision-and-Language Navigation in Continuous Environments (VLN-CE) typically relies on waypoint predictors to turn low-level robotic control into high-level viewpoint selection, ignoring low-level motion dynamics and physical obstacle semantics. This paper bridges this vision–action gap by introducing a dual-action framework—jointly predicting high-level viewpoints and autoregressively decoding low-level action sequences—alongside an obstacle-aware waypoint predictor enhanced with CLIP features and semantic obstacle masking.
What Problem Does This Paper Address?
In realistic continuous environments, embodied agents must navigate using low-level motor primitives (e.g., turning $15^\circ$, moving forward $0.25\text{ m}$). Direct end-to-end training on low-level actions struggles because continuous trajectories are long (~56 steps vs. 4–6 in discrete graphs), compounding errors.
To cope, top VLN-CE methods use an offline waypoint predictor to discretize the local continuous space into candidate viewpoints, training the agent solely as a high-level view selector and handing physical execution over to a heuristic controller. However, this introduces two major gaps:
- The Vision–Action Gap: Agents trained purely on viewpoint selection bypass the spatial reasoning embedded in physical trajectories, resulting in weak grounding between instructions, observations, and physical motion.
- Semantic Obliviousness: Standard waypoint predictors rely on low-level visual features (e.g., ImageNet-pretrained ResNet) and ignore physical affordances (e.g., walls and tables cannot be traversed, whereas open floors and doorways can).
Methods
The authors tackle these issues with two complementary components:
┌────────────────────────┐
│ Panoramic Observations │
└───────────┬────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
┌────────────────────────────┐ ┌────────────────────────────┐
│ Obstacle-Aware Waypoint │ │ Multimodal Navigator │
│ Predictor │ │ (HAMT / ETPNav Backbone) │
│ - CLIP Vision Encoder │ └─────────────┬──────────────┘
│ - Semantic Obstacle Mask │ │
└──────────────┬─────────────┘ │
▼ ▼
Candidate Viewpoints [CLS] Representation
│ │
└───────────────┬───────────────┘
▼
┌─────────────────────┐
│ Dual-Action Module │
├─────────────────────┤
│ • High-Level Choice │
│ • Low-Level Decoder │
└─────────────────────┘
1. Obstacle-Aware Waypoint Predictor
- Semantic Visual Encoders: Replaces the standard ResNet-152 visual backbone with CLIP (ViT-B/16) to capture richer open-vocabulary semantics.
- Prior-Based Obstacle Masking: Utilizes MP3D semantic segmentation with a defined open-area vocabulary (
floor,stairs,doors). Areas outside this passability vocabulary are explicitly masked out before predicting the waypoint heatmap.
2. Dual-Action Navigator
Rather than choosing between high-level view selection or low-level classification:
- Autoregressive Text Decoder: Formulates low-level action generation as sequence generation. Conditioned on the navigator's multimodal
[CLS]token and a prompt (low actions:), a Transformer text decoder predicts a sequence of tokenized motor actions (e.g.,left, left, forward). - Joint Optimization: High-level view selection (trained with Imitation and Reinforcement Learning, $\mathcal{L}_{high}$) and low-level sequence generation (trained via cross-entropy loss, $\mathcal{L}_{low}$) are trained end-to-end: $$\mathcal{L} = \mathcal{L}_{high} + \mathcal{L}_{low}$$
Datasets
- R2R-CE (Room-to-Room in Continuous Environments): Built on the Habitat Simulator using 3D scans from Matterport3D (MP3D).
- Splits: 61 training scenes, 11 validation unseen scenes, and 18 test unseen scenes.
- Supervision: Waypoint connectivity graphs and ground-truth continuous paths mapped from discrete R2R trajectories.
Key Insights and Ideas
- Low-level supervision acts as a spatial regularizer: Forcing the navigator to explain how to physically reach a selected viewpoint improves the high-level viewpoint selection itself.
- Sequence generation outperforms per-step classification: Framing motor controls as an autoregressive string sequence resolves the optimization bottleneck of standard per-step low-level classifiers.
- Bigger vision models aren't always better for affordance: While CLIP significantly improved open-space waypoint prediction, video-pretrained foundation models like InternVideo actually deteriorated waypoint prediction accuracy (dropping
%Openfrom 0.80 to 0.65), showing that general video dynamics do not trivially translate to geometric traversability.
Key Results
- High-Level Navigation (R2R-CE Test Unseen):
- When integrated into WP-HAMT, Success Rate (SR) increased from 49% to 52%, and SPL increased from 0.45 to 0.47.
- On the graph-based ETPNav, SR improved from 55% to 56% (and up to 58% on Validation Unseen).
- Low-Level Navigation (R2R-CE Validation Unseen):
- Setting a new state-of-the-art for low-level execution on WP-HAMT, jumping from 35% SR / 0.32 SPL (standard classifier) to 44% SR / 0.38 SPL.
- On ETPNav + Ours, reached 48% SR and 0.42 SPL, outperforming prior low-level sequence-to-sequence models (e.g., WS-MGMap's 39% SR).
- Waypoint Predictor Ablation:
- Combining CLIP features and semantic masking increased the ratio of navigable waypoints placed in open space (
%Open) from 0.80 to 0.85, reducing Hausdorff distance ($d_H$) from 2.00 to 1.94.
Limitations and Future Work
- Computational Overhead: Training the autoregressive Transformer action decoder alongside the cross-modal navigator introduces non-trivial compute costs (training required 6 NVIDIA RTX GPUs for 2.5 days).
- Residual High-vs-Low Performance Gap: While narrowing the divide, a performance gap remains between executing with high-level teleports/controllers (54% SR on val unseen) versus decoded low-level controls (44% SR).
- Dataset Artifacts: The R2R-CE continuous trajectories were generated via shortest-turn path heuristics from discrete graphs. Consequently, decoded low-level actions can sometimes reach the intended target while technically disagreeing with linguistic turn instructions.
Summary updated