NDH-Full: Learning and Evaluating Navigational Agents on Full-Length Dialogue

Hyounghun Kim, Jialu Li, Mohit BansalEMNLP 2021 · OpenReview

Authors: Hyounghun Kim, Jialu Li, Mohit Bansal. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

Introduction

Vision-and-Language Navigation (VLN) requires agents to navigate through complex environments using natural language instructions. The Navigation from Dialogue History (NDH) task was introduced to model more realistic, multi-turn interactions between a human navigator and an oracle. However, recent analysis has revealed significant shortcomings in the standard NDH benchmark. Specifically, the task often provides fragmented supervision where sub-paths do not contain sufficient information to reach the goal. Furthermore, current metrics like Goal Progress (GP) can be "gamed" by agents that wander toward the target region without actually following instructions.

An illustration of the fragmented NDH task where a full path is divided into sub-paths based on dialogue turns.

The NDH-FULL Task

To address these issues, the authors propose NDH-FULL. Unlike the original NDH, which breaks navigation into small segments, NDH-FULL treats the entire dialogue and the complete trajectory as a single instance. This shift forces agents to ground long-range dependencies and understand the sequential nature of the conversation to reach the final goal correctly. The dataset is derived from the Cooperative Vision-and-Dialogue Navigation (CVDN) dataset, consisting of over 1,600 full-length paths.

Comparison between fragmented dialogue history in the original NDH and the full dialogue context in the proposed NDH-FULL.

Model Architecture

The researchers developed a state-of-the-art agent based on the LXMERT transformer architecture. A key innovation is the Dialogue Progressor module, which manages the agent's attention across the multi-turn conversation. By using a Visual Proxy Token to maintain a memory of past observations, the agent can decide when it has fulfilled instructions from a specific dialogue turn and should proceed to the next. The model is trained using a combination of Imitation Learning (IL) and Reinforcement Learning (RL).

Key Results and Insights

The study highlights the "wandering" problem: agents rewarded for global goal progress often fail at instruction following (low path fidelity), while those rewarded for local path following struggle to reach the goal. NDH-FULL mitigates this by providing the full context. The proposed model achieved a state-of-the-art Goal Progress of 5.51 on the standard NDH "Validation Unseen" leaderboard. On the new NDH-FULL task, the model sets a strong baseline, though the small performance gap between full-dialogue and no-dialogue baselines (11.12 vs 10.97 GP) underscores the task's extreme difficulty.

Conclusion and Future Work

NDH-FULL represents a more rigorous and realistic benchmark for dialogue-based navigation. While the LXMERT-based agent provides a powerful starting point, the task remains an open challenge. Future research should explore active perception—where agents can ask for clarification—and improved methods for long-term grounding in multi-turn dialogues to bridge the gap between current performance and human-level instruction following.

Summary updated

BibTeX
@inproceedings{kim2021ndh,
  author = {Hyounghun Kim and Jialu Li and Mohit Bansal},
  title = {{NDH-Full: Learning and Evaluating Navigational Agents on Full-Length Dialogue}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=_rB7qFOu43R},
}
Figure