CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations

Jialu Li, Hao Tan, Mohit Bansal

Introduction

Vision-and-Language Navigation (VLN) is a complex task where an autonomous agent must navigate a real-world environment based on natural language instructions. While significant progress has been made, two primary challenges persist: the reliance on monolingual (typically English) instructions and the tendency of agents to overfit to the low-level visual features—such as lighting and textures—of training environments.

In this paper, we introduce CLEAR (Cross-Lingual and Environment-Agnostic Representations), a framework designed to address these hurdles by learning robust, generalized representations for both language and vision. By aligning instructions across multiple languages and focusing on semantic objects rather than background textures, CLEAR achieves state-of-the-art performance on the Room-Across-Room (RxR) dataset and generalizes significantly better to unseen environments.

Methodology: The CLEAR Framework

The CLEAR framework operates in two distinct stages: representation learning followed by navigation training.

1. Representation Learning

The core innovation of CLEAR lies in how it prepares the agent to "understand" its inputs before it begins the navigation task.

  • Cross-Lingual Language Learning: Utilizing a multilingual BERT encoder, the model is trained to align instructions across three languages: English, Hindi, and Telugu. By ensuring that different language descriptions of the same physical path are mapped to similar vector spaces, the agent learns a universal semantic representation. This process leverages the complementary nature of different languages; for instance, a Telugu instruction might highlight a specific object that the English version omits, providing a richer grounding for the visual path.
  • Environment-Agnostic Visual Learning: To mitigate environment bias, we identify "semantically-aligned" paths—trajectories in different houses described by similar instructions. We use a pre-trained Mask R-CNN to ensure these image pairs contain the same semantic objects. The visual encoder is then trained to maximize the similarity between these images, effectively teaching the agent to prioritize high-level objects (e.g., a "sink" or "sofa") over environment-specific "decor" like tile patterns or wall colors.

2. Navigation Training

With these learned representations, the agent is trained on the navigation task using a hybrid approach. It employs Imitation Learning (IL) to follow expert trajectories and Reinforcement Learning (RL) to explore and optimize for the final goal.

Experimental Setup

We primarily evaluate CLEAR on the Room-Across-Room (RxR) dataset, a large-scale multilingual benchmark featuring 126,069 instructions. To test the robustness and transferability of our representations, we also conduct experiments on:

  • Room-to-Room (R2R): A standard English-only navigation task.
  • Cooperative Vision-and-Dialogue Navigation (CVDN): A task involving multi-turn dialogues between a "commander" and a "follower."

Results and Discussion

Our experiments demonstrate that CLEAR significantly improves navigation performance and generalization across all metrics.

  • State-of-the-Art Performance: On the RxR dataset, CLEAR achieved a 16.88% improvement in normalized Dynamic Time Warping (nDTW) over the strong baseline. Notably, it also outperformed a baseline utilizing CLIP—a powerful vision-language model—by 2.59% nDTW.
  • Enhanced Generalization: In "unseen" environments (houses not present during training), CLEAR improved the success rate on the R2R dataset by 5.3% compared to standard ResNet-based models. This suggests that the environment-agnostic visual training successfully reduced the agent's reliance on specific training textures.
  • Zero-Shot Transfer: The learned representations proved highly effective even without task-specific fine-tuning, showing strong zero-shot performance when transferred from RxR to R2R.

Conclusion

CLEAR demonstrates that cross-lingual alignment and object-centric visual representations are powerful tools for building robust VLN agents. By focusing on the shared semantic meaning across languages and the invariant properties of objects across environments, CLEAR moves us closer to agents that can navigate diverse, real-world settings. Future work will explore expanding the framework to more languages and addressing the dependencies on pre-trained object detectors.

Summary updated

Abstract

Vision-and-Language Navigation (VLN) tasks require an agent to navigate through the environment based on language instructions. In this paper, we aim to solve two key challenges in this task: utilizing multilingual instructions for improved instruction-path grounding and navigating through new environments that are unseen during training. To address these challenges, we propose CLEAR: Cross-Lingual and Environment-Agnostic Representations. First, our agent learns a shared and visually-aligned cross-lingual language representation for the three languages (English, Hindi and Telugu) in the Room-Across-Room dataset. Our language representation learning is guided by text pairs that are aligned by visual information. Second, our agent learns an environment-agnostic visual representation by maximizing the similarity between semantically-aligned image pairs (with constraints on object-matching) from different environments. Our environment agnostic visual representation can mitigate the environment bias induced by low-level visual information. Empirically, on the Room-Across-Room dataset, we show that our multilingual agent gets large improvements in all metrics over the strong baseline model when generalizing to unseen environments with the cross-lingual language representation and the environment-agnostic visual representation. Furthermore, we show that our learned language and visual representations can be successfully transferred to the Room-to-Room and Cooperative Vision-and-Dialogue Navigation task, and present detailed qualitative and quantitative generalization and grounding analysis. Our code is available at https://github.com/jialuli-luka/CLEAR
BibTeX
@inproceedings{li2022clear,
  author = {Jialu Li and Hao Tan and Mohit Bansal},
  title = {{CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=1x1Y8Cmmf1},
}
Figure