Multi-View Learning for Vision-and-Language Navigation

Qiaolin Xia, Xiujun Li, Chunyuan Li, Yonatan Bisk, Zhifang Sui, Jianfeng Gao, Yejin Choi, Noah A. SmitharXiv 2020 · arXiv

Abstract

Vision-and-Language Navigation (VLN) requires agents to follow natural language instructions in 3D environments. This task is often hampered by linguistic ambiguity and underspecification. We present LEO (Learn from EveryOne), a multi-view learning framework that trains agents to reason across multiple instructions for the same path. This approach significantly improves visual grounding and navigation robustness, yielding a 16% absolute increase in success rates on unseen environments.

Introduction: The Problem of Single-View Navigation

In VLN, an agent might be told to "Walk past the table and turn left at the clock." However, human language is inherently messy—instructions can be ambiguous or omit crucial details. Most models treat instructions in isolation, a "single-view" approach that struggles with the high variability of natural language.

The Room-to-Room (R2R) dataset provides three unique instructions for every path. LEO leverages these "multiple teachers" to reduce uncertainty, helping the agent learn a more consistent mapping between language and the 3D environment.

The LEO Framework

LEO (Learn from EveryOne) introduces a multi-view training paradigm:

  1. Shared Encoding: Textual features are extracted from all available instructions using a shared LSTM.
  2. Memory-Attended Context: The agent attends to relevant instruction segments based on its navigation history.
  3. Aggregation: Information from multiple views is fused into a joint representation ($z_t$) via mean-pooling, max-pooling, or concatenation. Mean-pooling proved most effective for generalization.
A successful navigation trajectory where the agent correctly identifies the target location using the LEO framework.

By sharing parameters across instructions, the model learns that different terms (e.g., "hallway" vs. "corridor") refer to the same visual landmarks, effectively reducing the conditional entropy of the navigation task.

Results and Performance

The impact of multi-view learning is most evident in unseen environments. LEO improved a baseline seq2seq agent's Success Rate weighted by Path Length (SPL) from 25.3% to 41.4%.

A failed navigation attempt illustrating the complexity of the search space and the potential for the agent to become lost without robust grounding.

Furthermore, LEO is model-agnostic. When integrated with state-of-the-art models like SMNA and EnvDrop, it provided absolute gains of 9% and 12%, respectively. The top-performing LEO+ model reached an SPL of 62% on the R2R benchmark.

Conclusion

LEO demonstrates that training with multiple instructions—even if only one is available at test time—forces the model to learn deeper, more generalized groundings. While the framework currently relies on multiple instructions during training, future work may explore how to generate synthetic "views" to further enhance navigation in data-scarce scenarios.

Summary updated

Abstract

Learning to navigate in a visual environment following natural language instructions is a challenging task because natural language instructions are highly variable, ambiguous, and under-specified. In this paper, we present a novel training paradigm, Learn from EveryOne (LEO), which leverages multiple instructions (as different views) for the same trajectory to resolve language ambiguity and improve generalization. By sharing parameters across instructions, our approach learns more effectively from limited training data and generalizes better in unseen environments. On the recent Room-to-Room (R2R) benchmark dataset, LEO achieves 16% improvement (absolute) over a greedy agent as the base agent (25.3% $\rightarrow$ 41.4%) in Success Rate weighted by Path Length (SPL). Further, LEO is complementary to most existing models for vision-and-language navigation, allowing for easy integration with the existing techniques, leading to LEO+, which creates the new state of the art, pushing the R2R benchmark to 62% (9% absolute improvement).
BibTeX
@misc{xia2020multi,
  author = {Qiaolin Xia and Xiujun Li and Chunyuan Li and Yonatan Bisk and Zhifang Sui and Jianfeng Gao and Yejin Choi and Noah A. Smith},
  title = {{Multi-View Learning for Vision-and-Language Navigation}},
  year = {2020},
  eprint = {2003.00857},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.00857},
}
Figure