Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions

Qingbin Zeng, Qinglong Yang, Shunan Dong, Heming Du, Liang Zheng, Fengli Xu, Yong Li

Abstract

This paper considers a scenario in city navigation: an AI agent is provided with language descriptions of the goal location with respect to some well-known landmarks; By only observing the scene around, including recognizing landmarks and road network connections, the agent has to make decisions to navigate to the goal location without instructions. This problem is very challenging, because it requires the agent to establish self-position and acquire spatial representation of complex urban environments, where landmarks are often invisible. In the absence of navigation instructions, such abilities are vital for the agent to make high-quality decisions in long-range city navigation. With the emergent reasoning ability of large language models (LLMs), a tempting baseline is to prompt LLMs to "react" on each observation and make decisions accordingly. However, this baseline has very poor performance as the agent often repeatedly visits the same locations and makes short-sighted, inconsistent decisions. To address these issues, this paper introduces a novel agentic workflow featured by its abilities to perceive, reflect, and plan. We show the designed workflow significantly improves the navigation ability of the LLM agent compared with state-of-the-art baselines.

Introduction

Most existing autonomous navigation systems rely on step-by-step instructions (like GPS turn-by-turn) or pre-loaded maps. However, a truly intelligent agent should be able to navigate like a human: by looking at landmarks, understanding its relative position, and planning a route even when the destination isn't visible. Current LLMs struggle with this "goal-directed" navigation. When used in a simple "React" mode (observing and acting step-by-step), they tend to be short-sighted. They might walk toward a goal only to hit a dead end, or pace in circles because they lack a "cognitive map" of where they’ve already been. The challenge lies in establishing self-positioning and spatial reasoning in a complex urban environment where landmarks are frequently obscured.

The PReP Workflow

To overcome the limitations of reactive agents, we introduce the PReP (Perceive, Reflect, and Plan) workflow. This system integrates visual perception with human-like memory and strategic planning.

Perception

The agent uses a fine-tuned LLaVA-7B model to identify landmarks in street-view images. Crucially, it doesn't just name them; it estimates their approximate direction and distance. This information is then used to infer the agent's relative position to the final goal. Zero-shot LLaVA has poor landmark recognition for specific urban areas; fine-tuning with LoRA on just 5,000 landmark images boosted perception accuracy to near-human levels.

The PReP agent perceives landmarks and infers direction to the goal across various city environments.

Reflection and Memory

To solve the "short-sightedness" problem, the agent maintains two types of memory:

  • Episodic Memory: A log of past movements and observations.
  • Semantic Memory: A high-level summary that acts as an "internal city map," helping the agent understand the environment's layout and develop strategies like taking detours.
  • Working Memory: A buffer using an "anticipate-evaluate" mechanism to maintain goal direction even when landmarks temporarily disappear from view.

Planning

Instead of making one-off moves, the agent generates long-term plans broken into sub-goals (e.g., "move east until the next intersection"). It updates these plans dynamically based on new reflections and road connectivity data.

Experimental Setup

We introduced a new benchmark based on real-world data from Central Business Districts (CBDs) in four global cities: Beijing, Shanghai, New York, and Paris. Each city dataset contains approximately 1,000 nodes and 2,500 street-view images. The environment is modeled as an undirected graph where nodes are 50 meters apart. Unlike datasets that provide step-by-step instructions, this benchmark only provides the starting point and a textual description of the goal's location relative to landmarks.

Results and Discussion

The PReP workflow achieved dominant results across all tested cities, reaching an average Success Rate (SR) of 54%. In Beijing, it achieved 66.68%, compared to just 41% for the standard "React" baseline and 45% for DEPS (a state-of-the-art planning baseline).

Comparison of navigation paths: PReP avoids the loops and dead ends that plague reactive agents.

Key insights from our results include:

  • Reflection is Critical: In ablation studies, removing the "Reflection" (memory) component caused a much larger performance drop than removing the "Planning" component. This suggests that for LLMs, knowing where you have been is more critical for navigation than just deciding where to go next.
  • Efficiency: The Success weighted by Path Length (SPL) was significantly higher for PReP, indicating it finds goals using relatively efficient routes rather than wandering.
  • Robustness: The agent's success rate remained stable even as the goal distance increased up to 2 kilometers, proving it can handle long-range navigation without losing its way.

Conclusion and Future Work

While PReP significantly advances instruction-free navigation, limitations remain. Peak performance currently relies on powerful closed-source models like GPT-4; open-source variants like LLaMA3-8B show promise but still trail in spatial reasoning. Furthermore, inference speed—currently about 12 seconds per step—must be optimized for real-time applications. Future work will explore more effective fine-tuning strategies for open-source models to close the gap with proprietary LLMs.

Summary updated

Abstract

This paper considers a scenario in city navigation: an AI agent is provided with language descriptions of the goal location with respect to some well-known landmarks; By only observing the scene around, including recognizing landmarks and road network connections, the agent has to make decisions to navigate to the goal location without instructions. This problem is very challenging, because it requires agent to establish self-position and acquire spatial representation of complex urban environment, where landmarks are often invisible. In the absence of navigation instructions, such abilities are vital for the agent to make high-quality decisions in long-range city navigation. With the emergent reasoning ability of large language models (LLMs), a tempting baseline is to prompt LLMs to "react" on each observation and make decisions accordingly. However, this baseline has very poor performance that the agent often repeatedly visits same locations and make short-sighted, inconsistent decisions. To address these issues, this paper introduces a novel agentic workflow featured by its abilities to perceive, reflect and plan. Specifically, we find LLaVA-7B can be fine-tuned to perceive the direction and distance of landmarks with sufficient accuracy for city navigation. Moreover, reflection is achieved through a memory mechanism, where past experiences are stored and can be retrieved with current perception for effective decision argumentation. Planning uses reflection results to produce long-term plans, which can avoid short-sighted decisions in long-range navigation. We show the designed workflow significantly improves navigation ability of the LLM agent compared with the state-of-the-art baselines.
BibTeX
@misc{zeng2024perceive,
  author = {Qingbin Zeng and Qinglong Yang and Shunan Dong and Heming Du and Liang Zheng and Fengli Xu and Yong Li},
  title = {{Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions}},
  year = {2024},
  eprint = {2408.04168},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.04168},
}
Figure