Introduction
Vision-and-Language Navigation (VLN) is a complex task where an autonomous agent must navigate a real-world environment based on natural language instructions. While significant progress has been made, two primary challenges persist: the reliance on monolingual (typically English) instructions and the tendency of agents to overfit to the low-level visual features—such as lighting and textures—of training environments.
In this paper, we introduce CLEAR (Cross-Lingual and Environment-Agnostic Representations), a framework designed to address these hurdles by learning robust, generalized representations for both language and vision. By aligning instructions across multiple languages and focusing on semantic objects rather than background textures, CLEAR achieves state-of-the-art performance on the Room-Across-Room (RxR) dataset and generalizes significantly better to unseen environments.
Methodology: The CLEAR Framework
The CLEAR framework operates in two distinct stages: representation learning followed by navigation training.
1. Representation Learning
The core innovation of CLEAR lies in how it prepares the agent to "understand" its inputs before it begins the navigation task.
- Cross-Lingual Language Learning: Utilizing a multilingual BERT encoder, the model is trained to align instructions across three languages: English, Hindi, and Telugu. By ensuring that different language descriptions of the same physical path are mapped to similar vector spaces, the agent learns a universal semantic representation. This process leverages the complementary nature of different languages; for instance, a Telugu instruction might highlight a specific object that the English version omits, providing a richer grounding for the visual path.
- Environment-Agnostic Visual Learning: To mitigate environment bias, we identify "semantically-aligned" paths—trajectories in different houses described by similar instructions. We use a pre-trained Mask R-CNN to ensure these image pairs contain the same semantic objects. The visual encoder is then trained to maximize the similarity between these images, effectively teaching the agent to prioritize high-level objects (e.g., a "sink" or "sofa") over environment-specific "decor" like tile patterns or wall colors.
2. Navigation Training
With these learned representations, the agent is trained on the navigation task using a hybrid approach. It employs Imitation Learning (IL) to follow expert trajectories and Reinforcement Learning (RL) to explore and optimize for the final goal.
Experimental Setup
We primarily evaluate CLEAR on the Room-Across-Room (RxR) dataset, a large-scale multilingual benchmark featuring 126,069 instructions. To test the robustness and transferability of our representations, we also conduct experiments on:
- Room-to-Room (R2R): A standard English-only navigation task.
- Cooperative Vision-and-Dialogue Navigation (CVDN): A task involving multi-turn dialogues between a "commander" and a "follower."
Results and Discussion
Our experiments demonstrate that CLEAR significantly improves navigation performance and generalization across all metrics.
- State-of-the-Art Performance: On the RxR dataset, CLEAR achieved a 16.88% improvement in normalized Dynamic Time Warping (nDTW) over the strong baseline. Notably, it also outperformed a baseline utilizing CLIP—a powerful vision-language model—by 2.59% nDTW.
- Enhanced Generalization: In "unseen" environments (houses not present during training), CLEAR improved the success rate on the R2R dataset by 5.3% compared to standard ResNet-based models. This suggests that the environment-agnostic visual training successfully reduced the agent's reliance on specific training textures.
- Zero-Shot Transfer: The learned representations proved highly effective even without task-specific fine-tuning, showing strong zero-shot performance when transferred from RxR to R2R.
Conclusion
CLEAR demonstrates that cross-lingual alignment and object-centric visual representations are powerful tools for building robust VLN agents. By focusing on the shared semantic meaning across languages and the invariant properties of objects across environments, CLEAR moves us closer to agents that can navigate diverse, real-world settings. Future work will explore expanding the framework to more languages and addressing the dependencies on pre-trained object detectors.
Summary updated