VLN-Trans: Translator for the Vision and Language Navigation Agent

Yue Zhang, Parisa Kordjamshidi

Abstract

Language understanding is essential for navigation agents to follow instructions effectively. However, two primary issues often hinder performance: landmarks mentioned in instructions may not be recognizable by the agent due to differing visual capabilities between humans and models, and landmarks may not be distinctive enough to help the agent choose between multiple candidate viewpoints. To address these challenges, we introduce VLN-Trans, a translator module designed to convert original human instructions into easy-to-follow, agent-centric sub-instruction representations at each navigation step. By focusing on recognizable and distinctive landmarks, and utilizing a new synthetic sub-instruction dataset (SyFiS), our approach achieves state-of-the-art results on the Room2Room (R2R), Room4Room (R4R), and R2R-Last benchmarks.

Introduction

Vision-and-Language Navigation (VLN) requires an autonomous agent to navigate through a 3D environment based on natural language directions. While humans can easily interpret instructions like "Walk past the sofa and turn right at the kitchen," agents often struggle with the "grounding" process—mapping these words to their visual sensors.

Existing models frequently fail because of a fundamental misalignment between human descriptions and robot perception. VLN-Trans addresses this by acting as an intermediary, translating complex human intent into a simplified "visual vocabulary" that the agent can reliably act upon.

The Grounding Challenge

The authors identify two major hurdles in current VLN tasks:

  1. Unrecognizable Landmarks: Humans use high-level semantic concepts (e.g., "the dining area"). If an agent's vision system is trained on low-level objects (e.g., "table," "chair"), it may fail to ground the high-level concept to its current view.
  2. Non-distinctive Landmarks: Instructions often use ambiguous terms. If an instruction says "enter the door" but the agent sees three identical doors, the instruction provides no utility for decision-making.

The VLN-Trans Framework

VLN-Trans integrates a dedicated translator module into a standard navigation backbone (specifically VLN↻BERT). Rather than processing the entire instruction as a static string, the translator adaptively generates a sub-instruction representation at every step based on the agent's current visual surroundings.

The SyFiS Dataset

To train the translator, the researchers developed the Synthetic Fine-grained Sub-instruction dataset (SyFiS). Using CLIP (Contrastive Language-Image Pre-training), they scanned candidate viewpoints to identify objects and generated synthetic instructions following the template: [Motion Verb] + [Landmark]. Crucially, the dataset prioritizes "distinctive" landmarks—objects that appear only in the correct target view and are absent from incorrect candidate views.

Training Objectives

The translator is pre-trained using three specialized tasks:

  • Sub-instruction Generation (SIG): The model learns to predict the correct synthetic sub-instruction for a given visual observation.
  • Distinctive Sub-instruction Learning (DSL): Utilizing a triplet loss, the model ensures that the representation of a "correct" instruction (containing a distinctive landmark) is mathematically closer to the visual target than vague or incorrect instructions.
  • Sub-instruction Split (SS): This auxiliary task helps the model identify which specific segment of a long, complex human instruction is relevant to the current navigation step.

Experimental Results

VLN-Trans was evaluated on several key benchmarks, demonstrating superior performance even when using standard ResNet-152 visual features:

  • Room2Room (R2R): The model achieved a 69% Success Rate (SR) on the "Val Unseen" set. This outperforms previous state-of-the-art models like HAMT, which rely on more computationally expensive Vision Transformer (ViT) features.
  • R2R-Last: In this "destination-only" challenge where the agent only receives the final goal description, VLN-Trans showed a 5% improvement in Success Rate. This suggests the translator is highly effective at inferring necessary intermediate steps.
  • Room4Room (R4R): The model maintained robust performance on this benchmark, which features significantly longer paths and more complex instructions.

Key Insights

  • Agent-Centric Translation: The core success of VLN-Trans lies in its ability to translate human intent into the agent's own visual capabilities. It doesn't just "read" the instruction; it reinterprets it based on what it can actually see.
  • The Power of Distinctiveness: By explicitly training the model to seek out landmarks that differentiate the correct path from incorrect ones, the authors significantly reduce navigation ambiguity.
  • Synthetic Data Utility: The project demonstrates that high-quality synthetic data, generated by models like CLIP, can effectively bridge the gap in "common sense" reasoning without requiring massive new human-annotated datasets.

Conclusion and Future Work

VLN-Trans represents a significant step forward in making navigation agents more reliable. While the current translator outputs a mathematical vector rather than human-readable text—limiting direct interpretability—its performance gains are undeniable. Future research may focus on integrating more advanced visual backbones like ViT or CLIP directly into the navigation loop and testing these systems in dynamic, real-world robotic environments beyond static simulators.

Summary updated

Abstract

Language understanding is essential for the navigation agent to follow instructions. We observe two kinds of issues in the instructions that can make the navigation task challenging: 1. The mentioned landmarks are not recognizable by the navigation agent due to the different vision abilities of the instructor and the modeled agent. 2. The mentioned landmarks are applicable to multiple targets, thus not distinctive for selecting the target among the candidate viewpoints. To deal with these issues, we design a translator module for the navigation agent to convert the original instructions into easy-to-follow sub-instruction representations at each step. The translator needs to focus on the recognizable and distinctive landmarks based on the agent's visual abilities and the observed visual environment. To achieve this goal, we create a new synthetic sub-instruction dataset and design specific tasks to train the translator and the navigation agent. We evaluate our approach on Room2Room~(R2R), Room4room~(R4R), and Room2Room Last (R2R-Last) datasets and achieve state-of-the-art results on multiple benchmarks.
BibTeX
@inproceedings{zhang2023vln,
  author = {Yue Zhang and Parisa Kordjamshidi},
  title = {{VLN-Trans: Translator for the Vision and Language Navigation Agent}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=IAIBeK8AFD},
}
Figure