Towards Coarse-grained Visual Language Navigation Task Planning Enhanced by Event Knowledge Graph

Kaichen Zhao, Yaoxian Song, Haiquan Zhao, Haoyu Liu, Tiefeng Li, Zhixu Li

Introduction

Visual Language Navigation (VLN) represents a critical frontier in robotics and artificial intelligence, requiring agents to navigate complex environments based on natural language instructions. Traditionally, VLN research has relied on "fine-grained" instructions—detailed, step-by-step directions that guide an agent through every turn and landmark. However, human communication is naturally "coarse-grained," favoring abstract goals like "go to the refrigerator" over granular movement commands.

This shift toward coarse-grained instructions introduces an "abstraction gap." While Large Language Models (LLMs) possess significant general knowledge, they often lack the domain-specific "event logic" required by simulated environments. For example, an LLM may understand the concept of cleaning an object but may not know the specific sequence of subtasks—such as finding a sink, picking up the object, and toggling a faucet—required by a specific simulator. To bridge this gap, researchers have developed EventNav, a framework that enhances task planning through a specialized Event Knowledge Graph.

The EventNav Framework

EventNav operates on a two-pronged approach designed to ground abstract planning in realistic environmental constraints.

1. VLN-EventKG: Domain-Specific Knowledge

The foundation of the framework is the VLN-EventKG, a specialized Event Knowledge Graph. By extracting over 150,000 nodes and 120,000 relationships from established VLN datasets (R2R, REVERIE, and ALFRED), the researchers created a repository of successful subtask sequences. This graph serves as a "scenery-specific" prior, constraining the LLM’s planning capabilities to actions that are actually executable within the simulator, thereby reducing the noise often introduced by general-purpose common sense.

2. Large-Small Model Collaboration

EventNav utilizes a collaborative architecture that separates high-level reasoning from low-level execution:

  • The Outer Loop (LLM Planner): This high-level planner processes coarse instructions and visual captions (generated via BLIP-2). By querying the VLN-EventKG, the LLM predicts the next logical subtask required to fulfill the abstract goal.
  • The Inner Loop (Small Model Executor): A transformer-based model handles the low-level actions, such as moving forward, turning, or interacting with objects, to complete the subtask defined by the Outer Loop.

Mitigating Error Accumulation: Dynamic History Backtracking

A primary challenge in long-sequence navigation is error accumulation; a single mistake early in a mission often leads to total failure. EventNav addresses this through a Dynamic History Backtracking mechanism. The small model generates two critical feedback signals: S (a binary indicator of subtask completion) and R (the probability of success).

If the success probability R drops significantly or exhibits a downward trend, the system triggers a backtrack. This forces the LLM to re-evaluate the current state and re-plan the subtask from a previous successful state, effectively allowing the agent to "realize" it is stuck and attempt a different path.

Experimental Results and Insights

The EventNav framework was evaluated across three major benchmarks: R2R (navigation), REVERIE (object localization), and ALFRED (complex interaction). The results demonstrate a consistent improvement over existing state-of-the-art models that utilize coarse-grained instructions:

  • Success Rate (SR): EventNav achieved an average improvement of over 5% across all benchmarks.
  • REVERIE: In "Unseen" environments, the model reached a 34% Success Rate, a substantial increase from the 22% achieved by previous methods like CKR.
  • ALFRED: On this highly complex interactive dataset, EventNav achieved a 10% Success Rate on unseen tasks, outperforming the previous best (EmBert) by 3%.

Beyond raw metrics, the research highlights the benefits of cross-domain knowledge fusion. By integrating data from different datasets, the model showed that "event logic"—the fundamental way humans move through and interact with domestic spaces—is transferable across different simulation environments.

Future Directions

While EventNav represents a significant step forward in bridging the abstraction gap, challenges remain. Long-horizon tasks still suffer from information loss, partly due to the system's reliance on text-based captions to describe visual scenes. Future work will likely focus on more direct "vision-to-planner" integration and expanding the VLN-EventKG to include a wider variety of edge cases, further refining the agent's ability to navigate the complexities of the real world.

Summary updated

BibTeX
@inproceedings{zhao2024towards,
  author = {Kaichen Zhao and Yaoxian Song and Haiquan Zhao and Haoyu Liu and Tiefeng Li and Zhixu Li},
  title = {{Towards Coarse-grained Visual Language Navigation Task Planning Enhanced by Event Knowledge Graph}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2024},
  url = {https://openreview.net/forum?id=OOzTswY2w8},
}
Figure