Empowering Robotics with Large Language Models: osmAG Map Comprehension with LLMs

Fujing Xie, Sören SchwertfegerarXiv 2024 · arXiv · OpenReview

Abstract: Large Language Models (LLMs) have demonstrated great potential in robotic applications by providing essential general knowledge. Mobile robots rely on map comprehension for tasks like localization and navigation. In this paper, we explore enabling LLMs to comprehend the topology and hierarchy of Area Graph, a text-based hierarchical, topometric semantic map representation utilizing polygons to demark areas such as rooms or buildings. Our experiments demonstrate that with the right map representation, LLMs can effectively comprehend Area Graph's topology and hierarchy. After straightforward fine-tuning, the LLaMA2 models exceeded ChatGPT-3.5 in mastering these aspects. Our dataset, dataset generation code, fine-tuned LoRA adapters can be accessed at https://github.com/xiefujing/LLM-osmAG-Comprehension.

Introduction

Traditional robotic maps, such as 2D occupancy grids or 3D point clouds, are highly effective for low-level navigation but remain unintelligible to Large Language Models (LLMs). This creates a significant semantic gap: while a human can instantly understand a text-based notice about a hallway closure, a robot typically remains oblivious until it physically encounters the barrier. To address this, we propose using osmAG (Area Graph in OpenStreetMap format), a text-based hierarchical, topometric semantic map representation that allows robots to "read" and reason about their environment.

Methodology

The core of our approach is representing the physical world as a graph of areas (e.g., rooms, corridors) and passages (e.g., doors). Unlike pixel-based grids, osmAG provides a structured topological view that is more compatible with the token-based processing of LLMs.

Visualization of the Area Graph showing topological connections between rooms and passages.

Map Representation and Prompting

We evaluated three variants of the osmAG XML format to determine which structure best facilitates LLM reasoning. "Variant 2," which explicitly embeds connection information within each area's data tag, proved most effective by reducing the cognitive load required to link disparate elements. We also implemented multi-level prompting, ranging from basic task descriptions to detailed "chain-of-thought" examples.

Example of the osmAG XML format and the prompt structure used for LLM training.

Fine-Tuning with LoRA

Initial tests showed that base open-source models like LLaMA2 struggled with zero-shot map comprehension, often achieving success rates as low as 0.1. To overcome this, we utilized Low-Rank Adaptation (LoRA) to fine-tune LLaMA2-7B and 13B models on specialized topological datasets. This allowed the models to master the specific syntax of map topology while retaining their general reasoning capabilities.

Experimental Datasets

We developed five distinct datasets to rigorously train and evaluate our models:

  1. Topological Dataset 1: Standard layouts for basic pathfinding.
  2. Topological Dataset 2: A large-scale set of 12,520 randomized layouts for robust training.
  3. Generalization Dataset 3: Unseen map layouts to test the model's ability to learn general planning principles rather than memorizing specific routes.
  4. Hierarchical Dataset 4: Focused on organizational navigation (e.g., locating a specific office within a building hierarchy).
  5. General Knowledge Dataset 5: A control set of 20 general questions to ensure fine-tuning did not result in "catastrophic forgetting."

Key Results and Insights

Our experiments yielded several critical findings:

  • High Performance: Fine-tuned LLaMA2 models achieved over 90% success in map comprehension tasks, with the 13B variant maintaining a 0.97 success rate on entirely new, unseen layouts.
  • Competitive Edge: The fine-tuned open-source models consistently outperformed ChatGPT-3.5 in these specialized navigation tasks.
  • Metric-Free Reasoning: By focusing on topology (e.g., "Room A connects to Hallway B") rather than precise metric coordinates, we made spatial reasoning significantly more manageable for the LLMs.
  • Real-World Utility: In qualitative tests, the system successfully rerouted a robot around a simulated maintenance area after processing a text-based notification, demonstrating the practical value of combining LLMs with semantic maps.

Conclusion and Future Work

While LLMs show immense promise for semantic navigation, challenges remain regarding token limits for large-scale buildings and the mathematical optimality of paths compared to classical algorithms like A*. Future research will focus on hybrid architectures that leverage LLMs for high-level semantic reasoning and traditional algorithms for low-level geometric optimization.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated great potential in robotic applications by providing essential general knowledge. Mobile robots rely on map comprehension for tasks like localization and navigation. In this paper, we explore enabling LLMs to comprehend the topology and hierarchy of Area Graph, a text-based hierarchical, topometric semantic map representation utilizing polygons to demark areas such as rooms or buildings. Our experiments demonstrate that with the right map representation, LLMs can effectively comprehend Area Graph's topology and hierarchy. After straightforward fine-tuning, the LLaMA2 models exceeded ChatGPT-3.5 in mastering these aspects. Our dataset, dataset generation code, fine-tuned LoRA adapters can be accessed at https://github.com/xiefujing/LLM-osmAG-Comprehension.
BibTeX
@misc{xie2024empowering,
  author = {Fujing Xie and Sören Schwertfeger},
  title = {{Empowering Robotics with Large Language Models: osmAG Map Comprehension with LLMs}},
  year = {2024},
  eprint = {2403.08228},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.08228},
}
Figure