Abstract: The existing work on vision and language navigation mainly relies on navigation-related losses to establish the connection between vision and language modalities, neglecting aspects of helping the navigation agent build a deep understanding of the visual environment. In our work, we provide indirect supervision to the navigation agent through a hint generator that provides detailed visual descriptions. The hint generator assists the navigation agent in developing a global understanding of the visual environment. It directs the agent’s attention toward related navigation details, including the relevant sub-instruction, potential challenges in recognition and ambiguities in grounding, and the targeted viewpoint description. To train the hint generator, we construct a synthetic dataset based on landmarks in the instructions and visible and distinctive objects in the visual environment. We evaluate our method on the R2R and R4R datasets and achieve state-of-the-art on several metrics. The experimental results demonstrate that generating hints not only enhances the navigation performance but also helps improve the agent’s interpretability.
Introduction
Vision and Language Navigation (VLN) requires an agent to follow natural language instructions (e.g., "Walk past the table and stop by the sofa") within a 3D environment. Traditionally, these agents are trained using "wayfinding" signals—rewards based on goal proximity or path mimicry. However, this approach often fails to foster deep visual understanding. An agent might reach the goal by chance or by exploiting simple correlations without truly grounding the landmarks mentioned.
A major challenge is landmark ambiguity: if an instruction says "turn at the wall" but the agent sees multiple walls, standard models often struggle to differentiate between them. NavHint addresses this by introducing a "hint generator" that provides detailed, step-by-step visual descriptions, shifting the focus from simply "where to move" to a more comprehensive "what do you see?"
Methods
The core innovation is NavHint, a plug-and-play hint generator that can be integrated into existing VLN models. At each navigation step, the generator produces a text string containing three key elements:
- Sub-instruction: Pinpoints the specific part of the long instruction currently being executed.
- Landmark Ambiguity: Identifies if a landmark is unique to the target view, visible in multiple views (ambiguous), or missing entirely.
- Targeted Distinctive Objects: Describes objects unique to the chosen viewpoint that distinguish it from other candidates.
As shown in the architecture above, the generator uses a Transformer-based decoder (GPT-2 Large) conditioned on the agent's visual features and the instruction text. It employs "prefix tuning" to map visual representations into the language model's token space, prompting the generation of the hint.
Datasets and Training
The model was evaluated on the standard R2R (Room2Room) and R4R (Room-for-Room) benchmarks. To train the hint generator, the authors constructed a new Navigation Hint Dataset. They used spaCy to extract landmarks from instructions and MiniGPT-4 to generate detailed descriptions of objects in the Matterport3D environment. This synthetic dataset provides the ground truth hints necessary for training the generator to identify distinctive visual cues.
Key Results
NavHint achieves state-of-the-art performance on several key metrics:
- R2R Performance: On the "Test Unseen" split, NavHint achieved a Success Rate (SR) of 65% and an SPL (Success weighted by Path Length) of 0.60.
- Instruction Following: The model showed significant gains in nDTW (normalized Dynamic Time Warping), indicating that the agent's path more closely follows the intended route rather than just reaching the destination by luck.
- Interpretability: By generating text hints at every step, the agent's decision-making process becomes transparent. Researchers can diagnose failures by reviewing the generated hints to see if the agent misidentified landmarks or overlooked distinctive objects.
Conclusion
NavHint demonstrates that providing indirect supervision through descriptive hints significantly enhances both the performance and interpretability of VLN agents. While the current implementation uses GPT-2 and ResNet-152 features, future work could explore scaling to larger language models and modern Vision Transformers (ViT) to further refine the quality of generated hints and navigation accuracy.
Summary updated