Abstract
Vision-and-Language Navigation (VLN) requires agents to follow natural language instructions in 3D environments. This task is often hampered by linguistic ambiguity and underspecification. We present LEO (Learn from EveryOne), a multi-view learning framework that trains agents to reason across multiple instructions for the same path. This approach significantly improves visual grounding and navigation robustness, yielding a 16% absolute increase in success rates on unseen environments.
Introduction: The Problem of Single-View Navigation
In VLN, an agent might be told to "Walk past the table and turn left at the clock." However, human language is inherently messy—instructions can be ambiguous or omit crucial details. Most models treat instructions in isolation, a "single-view" approach that struggles with the high variability of natural language.
The Room-to-Room (R2R) dataset provides three unique instructions for every path. LEO leverages these "multiple teachers" to reduce uncertainty, helping the agent learn a more consistent mapping between language and the 3D environment.
The LEO Framework
LEO (Learn from EveryOne) introduces a multi-view training paradigm:
- Shared Encoding: Textual features are extracted from all available instructions using a shared LSTM.
- Memory-Attended Context: The agent attends to relevant instruction segments based on its navigation history.
- Aggregation: Information from multiple views is fused into a joint representation ($z_t$) via mean-pooling, max-pooling, or concatenation. Mean-pooling proved most effective for generalization.
By sharing parameters across instructions, the model learns that different terms (e.g., "hallway" vs. "corridor") refer to the same visual landmarks, effectively reducing the conditional entropy of the navigation task.
Results and Performance
The impact of multi-view learning is most evident in unseen environments. LEO improved a baseline seq2seq agent's Success Rate weighted by Path Length (SPL) from 25.3% to 41.4%.
Furthermore, LEO is model-agnostic. When integrated with state-of-the-art models like SMNA and EnvDrop, it provided absolute gains of 9% and 12%, respectively. The top-performing LEO+ model reached an SPL of 62% on the R2R benchmark.
Conclusion
LEO demonstrates that training with multiple instructions—even if only one is available at test time—forces the model to learn deeper, more generalized groundings. While the framework currently relies on multiple instructions during training, future work may explore how to generate synthetic "views" to further enhance navigation in data-scarce scenarios.
Summary updated