Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Based Zero-Shot Object Navigation

Vishnu Sashank Dorbala, James F. Mullen Jr., Dinesh Manocha

Abstract

We present LGX (Language-guided Exploration), a novel algorithm for Language-Driven Zero-Shot Object Goal Navigation (L-ZSON), where an embodied agent navigates to a uniquely described target object in a previously unseen environment. Our approach makes use of Large Language Models (LLMs) for this task by leveraging the LLM's commonsense reasoning capabilities for making sequential navigational decisions. Simultaneously, we perform generalized target object detection using a pre-trained Vision-Language grounding model. We achieve state-of-the-art zero-shot object navigation results on RoboTHOR with a success rate (SR) improvement of over 27% over the current baseline of the OWL-ViT CLIP on Wheels (OWL CoW). Furthermore, we study the usage of LLMs for robot navigation and present an analysis of various prompting strategies affecting the model output. Finally, we showcase the benefits of our approach via real-world experiments that indicate the superior performance of LGX in detecting and navigating to visually unique objects.

Introduction

Traditional Object Goal Navigation (ObjectNav) typically restricts robots to finding a predefined set of object categories, such as "chair" or "bed." However, real-world human-robot interaction requires a more nuanced understanding of natural language. A user might ask for a "blue pillow" or a "cat-shaped mug"—targets that fall outside standard datasets.

Overview of the LGX framework navigating to a 'cat-shaped mug' by combining LLM decision-making with vision-language grounding.

To address this, we introduce LGX (Language-Guided Exploration), a framework for Language-Driven Zero-Shot Object Goal Navigation (L-ZSON). LGX moves beyond "uninformed" exploration strategies, like Frontier-Based Exploration (FBE), by leveraging the semantic common sense of Large Language Models (LLMs) to guide the robot toward likely locations of uniquely described objects.

The LGX Framework

LGX decomposes the navigation task into two concurrent processes: sequential decision-making and target grounding.

Detailed system architecture of LGX, illustrating the flow from observation to motion planning.

1. Scene Understanding and Semantic Extraction

At each timestep, the agent performs a 360-degree scan of its environment. It uses zero-shot object detection (YOLO) or image captioning (BLIP) to identify surrounding landmarks. This semantic information is converted into a text-based prompt.

2. LLM Decision-Making

The prompt, containing the target description and the list of observed objects, is fed into an LLM (e.g., GPT-3). The LLM acts as a "semantic compass," using its internal world knowledge to suggest which observed object the agent should move toward. For instance, if looking for a "mug" and seeing a "table" and a "bed," the LLM will prioritize the table.

3. Target Grounding with GLIP

Simultaneously, the system uses GLIP (Grounded Language-Image Pre-training) to scan the visual feed for the specific target. Unlike standard detectors, GLIP can handle descriptive modifiers, allowing it to distinguish a "cat-shaped mug" from a standard mug.

4. Motion Planning

Once a sub-goal is identified by the LLM or the target is grounded by GLIP, the agent utilizes standard ROS tools to build a local costmap and navigate safely while avoiding obstacles.

Experimental Results

The performance of LGX was evaluated in both the RoboTHOR simulation environment and real-world settings:

  • RoboTHOR Benchmark: LGX achieved a 35.0% Success Rate (SR), a significant improvement over the previous state-of-the-art baseline, OWL-CoW (27.5%). It also demonstrated higher efficiency in path length (SPL).
  • Real-World Validation: Using a TurtleBot 2, LGX successfully navigated to non-standard objects like "stevia sugar packets" and an "olive-colored jacket" in a multi-room environment, achieving a 54.2% success rate compared to just 6.9% for random exploration.

Key Insights

  • Prompting Strategies: The research highlights that providing a structured list of detected objects (YOLO → LLM) is more effective than natural language captions (BLIP → LLM). Placing the object list at the beginning of the prompt also improves reasoning.
  • Semantic Reasoning: LLMs effectively bridge the gap between high-level language and spatial reasoning, turning "uninformed" wandering into "informed" searching based on latent world knowledge.

Limitations and Future Work

While LGX shows great promise, it currently lacks long-term memory, which can lead to redundant exploration. Additionally, LLM "hallucinations" and the detection of very small or transparent objects remain challenges. Future iterations will focus on incorporating navigation history into prompts and utilizing more advanced grounding models like Detic to improve detection accuracy.

Summary updated

Abstract

We present LGX (Language-guided Exploration), a novel algorithm for Language-Driven Zero-Shot Object Goal Navigation (L-ZSON), where an embodied agent navigates to a uniquely described target object in a previously unseen environment. Our approach makes use of Large Language Models (LLMs) for this task by leveraging the LLM's commonsense reasoning capabilities for making sequential navigational decisions. Simultaneously, we perform generalized target object detection using a pre-trained Vision-Language grounding model. We achieve state-of-the-art zero-shot object navigation results on RoboTHOR with a success rate (SR) improvement of over 27% over the current baseline of the OWL-ViT CLIP on Wheels (OWL CoW). Furthermore, we study the usage of LLMs for robot navigation and present an analysis of various prompting strategies affecting the model output. Finally, we showcase the benefits of our approach via \textit{real-world} experiments that indicate the superior performance of LGX in detecting and navigating to visually unique objects.
BibTeX
@article{dorbala2024can,
  author = {Vishnu Sashank Dorbala and James F. Mullen Jr. and Dinesh Manocha},
  title = {{Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Based Zero-Shot Object Navigation}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
  doi = {10.1109/lra.2023.3346800},
}
Figure