CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations

Jialu Li, Hao Tan, Mohit Bansal

Abstract

Vision-and-Language Navigation (VLN) tasks require an agent to navigate through complex 3D environments based on natural language instructions. This paper introduces CLEAR (Cross-Lingual, Environment-Agnostic Representations), a framework designed to address two critical challenges in VLN: grounding multilingual instructions and generalizing to unseen environments. By learning a shared cross-lingual representation for English, Hindi, and Telugu and an environment-agnostic visual representation through semantically-aligned image pairs, the CLEAR agent achieves significant performance gains. On the Room-Across-Room (RxR) dataset, CLEAR demonstrates superior generalization in unseen environments and successfully transfers to other tasks like Room-to-Room (R2R) and Cooperative Vision-and-Dialogue Navigation (CVDN).

Introduction

In Vision-and-Language Navigation, an autonomous agent must interpret human instructions (e.g., "Walk past the kitchen and stop by the sofa") to reach a goal in a 3D space. Despite recent progress, current agents suffer from two major limitations:

  1. Multilingual Fragility: Most models are optimized for English. In multilingual settings like the Room-Across-Room (RxR) dataset, agents struggle to map different languages to the same physical actions.
  2. The "Unseen" Gap: Agents often overfit to low-level visual textures (like specific wallpaper or lighting) of training environments. When placed in a new building with different decor, their performance drops significantly due to domain shift.

The CLEAR Framework

To overcome these hurdles, the authors propose a two-stage learning process that focuses on robust representation learning before navigation training.

Stage 1: Representation Learning

CLEAR utilizes contrastive learning to align language and vision in a way that transcends specific languages or room aesthetics.

  • Cross-Lingual Alignment: The model is presented with instruction pairs in different languages (English, Hindi, Telugu) that describe the same physical path. By forcing the embeddings of these pairs to be close in a shared space, the model learns that terms like "wash basin" (English) and "వాష్ బేసిన్" (Telugu) refer to the same visual concept.
  • Environment-Agnostic Vision: To prevent the agent from obsessing over environment-specific details, the framework identifies paths in different buildings that share similar instructions. Using an Object-Matching Constraint (via Mask-RCNN), the agent learns to recognize high-level landmarks (e.g., a "chair" or "bathtub") regardless of the specific visual style of the room.
Multilingual instructions aligned with visual paths in different environments.

Stage 2: Navigation Training

With these robust encoders as a foundation, the agent is trained on the navigation task using a combination of Imitation Learning (following expert paths) and Reinforcement Learning (rewarding goal reaching and path fidelity).

Experimental Results

The CLEAR model was primarily evaluated on the Room-Across-Room (RxR) dataset, which features 126k instructions across three languages in Matterport3D environments.

  • State-of-the-Art Performance: CLEAR outperformed strong CLIP-based mono-lingual baselines by 2.59% in nDTW (normalized Dynamic Time Warping), the primary metric for measuring how closely an agent follows a prescribed path.
  • Enhanced Generalization: In "unseen" environments, CLEAR improved the Success Rate (SR) by 5.3% over standard ResNet-based models. The narrowed gap between "seen" and "unseen" performance indicates that the model is learning generalizable navigation skills rather than memorizing maps.
  • Zero-Shot Transfer: The representations learned on RxR successfully improved performance on the Room-to-Room (R2R) dataset by 4.5% in a zero-shot setting, demonstrating the versatility of the environment-agnostic approach.

Conclusion

CLEAR demonstrates that multilingualism can be a feature rather than a hurdle; by aligning multiple languages to a single visual path, agents develop a richer, more grounded understanding of their surroundings. Furthermore, by prioritizing high-level semantic landmarks over low-level visual noise, CLEAR provides a roadmap for building agents that can navigate the diverse and unpredictable environments of the real world. Future work will look toward scaling these representations to a broader range of languages and more granular object interactions.

Summary updated

Abstract

Vision-and-Language Navigation (VLN) tasks require an agent to navigate through the environment based on language instructions. In this paper, we aim to solve two key challenges in this task: utilizing multilingual instructions for improved instruction-path grounding and navigating through new environments that are unseen during training. To address these challenges, first, our agent learns a shared and visually-aligned cross-lingual language representation for the three languages (English, Hindi and Telugu) in the Room-Across-Room dataset. Our language representation learning is guided by text pairs that are aligned by visual information. Second, our agent learns an environment-agnostic visual representation by maximizing the similarity between semantically-aligned image pairs (with constraints on object-matching) from different environments. Our environment agnostic visual representation can mitigate the environment bias induced by low-level visual information. Empirically, on the Room-Across-Room dataset, we show that our multi-lingual agent gets large improvements in all metrics over the strong baseline model when generalizing to unseen environments with the cross-lingual language representation and the environment-agnostic visual representation. Furthermore, we show that our learned language and visual representations can be successfully transferred to the Room-to-Room and Cooperative Vision-and-Dialogue Navigation task, and present detailed qualitative and quantitative generalization and grounding analysis.
BibTeX
@inproceedings{li2022clear,
  author = {Jialu Li and Hao Tan and Mohit Bansal},
  title = {{CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=1x1Y8Cmmf1},
}
Figure