Abstract
Vision-and-Language Navigation (VLN) tasks require an agent to navigate through complex 3D environments based on natural language instructions. This paper introduces CLEAR (Cross-Lingual, Environment-Agnostic Representations), a framework designed to address two critical challenges in VLN: grounding multilingual instructions and generalizing to unseen environments. By learning a shared cross-lingual representation for English, Hindi, and Telugu and an environment-agnostic visual representation through semantically-aligned image pairs, the CLEAR agent achieves significant performance gains. On the Room-Across-Room (RxR) dataset, CLEAR demonstrates superior generalization in unseen environments and successfully transfers to other tasks like Room-to-Room (R2R) and Cooperative Vision-and-Dialogue Navigation (CVDN).
Introduction
In Vision-and-Language Navigation, an autonomous agent must interpret human instructions (e.g., "Walk past the kitchen and stop by the sofa") to reach a goal in a 3D space. Despite recent progress, current agents suffer from two major limitations:
- Multilingual Fragility: Most models are optimized for English. In multilingual settings like the Room-Across-Room (RxR) dataset, agents struggle to map different languages to the same physical actions.
- The "Unseen" Gap: Agents often overfit to low-level visual textures (like specific wallpaper or lighting) of training environments. When placed in a new building with different decor, their performance drops significantly due to domain shift.
The CLEAR Framework
To overcome these hurdles, the authors propose a two-stage learning process that focuses on robust representation learning before navigation training.
Stage 1: Representation Learning
CLEAR utilizes contrastive learning to align language and vision in a way that transcends specific languages or room aesthetics.
- Cross-Lingual Alignment: The model is presented with instruction pairs in different languages (English, Hindi, Telugu) that describe the same physical path. By forcing the embeddings of these pairs to be close in a shared space, the model learns that terms like "wash basin" (English) and "వాష్ బేసిన్" (Telugu) refer to the same visual concept.
- Environment-Agnostic Vision: To prevent the agent from obsessing over environment-specific details, the framework identifies paths in different buildings that share similar instructions. Using an Object-Matching Constraint (via Mask-RCNN), the agent learns to recognize high-level landmarks (e.g., a "chair" or "bathtub") regardless of the specific visual style of the room.
Stage 2: Navigation Training
With these robust encoders as a foundation, the agent is trained on the navigation task using a combination of Imitation Learning (following expert paths) and Reinforcement Learning (rewarding goal reaching and path fidelity).
Experimental Results
The CLEAR model was primarily evaluated on the Room-Across-Room (RxR) dataset, which features 126k instructions across three languages in Matterport3D environments.
- State-of-the-Art Performance: CLEAR outperformed strong CLIP-based mono-lingual baselines by 2.59% in nDTW (normalized Dynamic Time Warping), the primary metric for measuring how closely an agent follows a prescribed path.
- Enhanced Generalization: In "unseen" environments, CLEAR improved the Success Rate (SR) by 5.3% over standard ResNet-based models. The narrowed gap between "seen" and "unseen" performance indicates that the model is learning generalizable navigation skills rather than memorizing maps.
- Zero-Shot Transfer: The representations learned on RxR successfully improved performance on the Room-to-Room (R2R) dataset by 4.5% in a zero-shot setting, demonstrating the versatility of the environment-agnostic approach.
Conclusion
CLEAR demonstrates that multilingualism can be a feature rather than a hurdle; by aligning multiple languages to a single visual path, agents develop a richer, more grounded understanding of their surroundings. Furthermore, by prioritizing high-level semantic landmarks over low-level visual noise, CLEAR provides a roadmap for building agents that can navigate the diverse and unpredictable environments of the real world. Future work will look toward scaling these representations to a broader range of languages and more granular object interactions.
Summary updated