VLN-Trans: Translator for the Vision and Language Navigation Agent

Yue Zhang, Parisa Kordjamshidi

Abstract

Language understanding is essential for the navigation agent to follow instructions. We observe two kinds of issues in the instructions that can make the navigation task challenging:

  1. The mentioned landmarks are not recognizable by the navigation agent due to the different vision abilities of the instructor and the modeled agent.
  2. The mentioned landmarks are applicable to multiple targets, thus not distinctive for selecting the target among the candidate viewpoints. To deal with these issues, we design a translator module for the navigation agent to convert the original instructions into easy-to-follow sub-instruction representations at each step. The translator needs to focus on the recognizable and distinctive landmarks based on the agent’s visual abilities and the observed visual environment. To achieve this goal, we create a new synthetic sub-instruction dataset and design specific tasks to train the translator and the navigation agent. We evaluate our approach on Room2Room (R2R), Room4room (R4R), and Room2Room Last (R2R-Last) datasets and achieve state-of-the-art results on multiple benchmarks.

Introduction

Vision-and-Language Navigation (VLN) tasks an agent with reaching a destination in a photo-realistic environment by following natural language instructions. While significant progress has been made, agents still struggle with the discrepancy between human-provided instructions and their own visual perception. This paper identifies two core challenges that hinder effective navigation:

  1. The Recognition Gap: Humans often use high-level semantic landmarks (e.g., "the kitchen") that require prior knowledge. Agents, however, perceive low-level objects (e.g., "a toaster") and may fail to ground the high-level concept into their visual reality.
  2. The Ambiguity Problem: Instructions are often non-distinctive. A command like "enter the door" is insufficient if the agent's current view contains multiple doors.
Figure 1: Illustration of the Recognition Gap (a) and the Ambiguity Problem (b) in standard VLN instructions.

The VLN-Trans Framework

To bridge this gap, we propose VLN-Trans, which incorporates a dedicated Translator Module into a standard navigation backbone (VLN⟳BERT). Instead of forcing the agent to interpret raw, complex instructions directly, the translator converts them into simplified, step-by-step "sub-instruction representations."

The Translator Module

The translator takes the full instruction and the agent's current visual observations as input. It generates a high-dimensional vector representation that highlights the most relevant and distinctive landmarks for the immediate next step. This allows the navigation agent to focus on actionable visual cues rather than abstract human prior knowledge.

Synthetic Fine-grained Sub-instruction Dataset (SyFiS)

Training such a translator requires fine-grained data. We created SyFiS, a synthetic dataset of over 1 million trajectories. We utilized CLIP to identify "recognizable" objects in a scene and filtered them to find "distinctive" landmarks—objects that appear uniquely in the correct target viewpoint compared to distracting alternative paths.

Training Objectives

The framework is trained using three specialized tasks:

  • Sub-instruction Generation (SG): Learning to predict the correct sub-instruction for a given viewpoint.
  • Distinctive Sub-instruction Learning (DSL): A contrastive triplet loss approach that encourages the model to favor distinctive landmarks over ambiguous ones by comparing anchors to positive and negative samples.
  • Sub-instruction Split (SS): An auxiliary task that helps the model identify which parts of a long instruction correspond to the current navigation step.
Figure 5: The VLN-Trans architecture, showing the interaction between the Translator and the Navigation Agent (a), the Sub-instruction Generation process (b), and the Contrastive Learning framework (c).

Experimental Results

We evaluated VLN-Trans on three major benchmarks: Room2Room (R2R), Room4Room (R4R), and Room2Room Last (R2R-Last).

  • R2R Unseen: VLN-Trans achieved a Success Rate (SR) of 69% and a Success weighted by Path Length (SPL) of 63%, setting a new state-of-the-art and outperforming previous models like HAMT.
  • R2R-Last: On this difficult "destination-only" task, the model improved the SR by nearly 5% over previous best models, demonstrating the translator's ability to extract goal-relevant cues from limited text.
  • R4R: The model showed superior path fidelity, accurately following long, complex trajectories more closely than baseline models.

Conclusion

VLN-Trans addresses the fundamental mismatch between human language and robot vision by introducing a translator that prioritizes recognizable and distinctive landmarks. By shifting from raw text to optimized sub-instruction representations, the agent can navigate more effectively in ambiguous environments. Future work will focus on improving the interpretability of the translator's representations and transitioning the system from simulation to real-world robotic hardware.

Summary updated

Abstract

Language understanding is essential for the navigation agent to follow instructions. We observe two kinds of issues in the instructions that can make the navigation task challenging: 1. The mentioned landmarks are not recognizable by the navigation agent due to the different vision abilities of the instructor and the modeled agent. 2. The mentioned landmarks are applicable to multiple targets, thus not distinctive for selecting the target among the candidate viewpoints. To deal with these issues, we design a translator module for the navigation agent to convert the original instructions into easy-to-follow sub-instruction representations at each step. The translator needs to focus on the recognizable and distinctive landmarks based on the agent’s visual abilities and the observed visual environment. To achieve this goal, we create a new synthetic sub-instruction dataset and design specific tasks to train the translator and the navigation agent. We evaluate our approach on Room2Room (R2R), Room4room (R4R), and Room2Room Last (R2R-Last) datasets and achieve state-of-the-art results on multiple benchmarks.
BibTeX
@inproceedings{zhang2023vln,
  author = {Yue Zhang and Parisa Kordjamshidi},
  title = {{VLN-Trans: Translator for the Vision and Language Navigation Agent}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=IAIBeK8AFD},
}
Figure