Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Based Zero-Shot Object Navigation

Vishnu Sashank Dorbala, James F. Mullen Jr., Dinesh Manocha

TL;DR

LGX (Language-Guided Exploration) is a new framework that enables robots to find objects described in free-form natural language—like a "cat-shaped mug"—in completely unfamiliar environments. By leveraging the "commonsense" reasoning of Large Language Models (LLMs) to make navigational decisions and vision-language models for detection, the system achieves a 27% improvement in success rates over previous state-of-the-art methods.

What Problem Does This Paper Address?

Traditional "Object Goal Navigation" (ObjectNav) is often a rigid affair. Robots are typically trained to find a fixed set of categories (e.g., "chair," "bed," "toilet"). However, humans don't talk like datasets; we might ask a robot to find a "blue pillow" or an "olive-colored jacket."

Existing "zero-shot" methods (which try to find objects the robot hasn't been specifically trained on) often rely on "uninformed" exploration. They might use Frontier-Based Exploration (FBE), where the robot simply moves toward the nearest unexplored open space. This is inefficient because it ignores semantic context—for example, a robot shouldn't look for a "remote control" in a bathroom if it has already seen a "couch" in the living room. The challenge is combining flexible language understanding with intelligent, semantically-aware exploration.

Methods

The authors propose LGX, which splits the navigation task into two concurrent processes: Sequential Decision Making and Target Grounding.

  1. Scene Understanding: At each step, the robot performs a 360-degree scan. It uses either YOLO (to list common objects) or BLIP (to generate image captions) to describe what it sees.
  2. LLM Decision Making: This text description, along with the user's target (e.g., "cat-shaped mug"), is fed into an LLM (GPT-3). The LLM uses its internal "world knowledge" to suggest the best direction. If the robot sees a table and a bed, and is looking for a mug, the LLM will suggest moving toward the table.
  3. Target Grounding: While the LLM decides where to go, a vision-language model called GLIP (Grounded Language-Image Pre-training) scans the camera feed for the specific target. Unlike standard detectors, GLIP can handle descriptive language, allowing it to distinguish a "cat-shaped mug" from a regular "mug" or a "cat."
  4. Motion Planning: The robot uses standard ROS (Robot Operating System) tools to build a local costmap and navigate safely to the LLM's suggested sub-goal while avoiding obstacles.

Datasets

The researchers evaluated LGX in both simulation and the real world:

  • RoboTHOR: A large-scale simulated environment for embodied AI. They used 1,800 validation episodes across 15 different house layouts.
  • Real-World Experiments: Conducted using a TurtleBot 2 platform in a multi-room setup (Office, Living Room, Bedroom, Kitchen). They tested unique, non-standard objects like a "Red Bull can," "stevia sugar packets," and an "olive-colored jacket" to prove the system's flexibility.

Key Insights and Ideas

  • LLMs as Semantic Compasses: The core "aha" moment is that LLMs already contain the spatial common sense needed for navigation. They know that "remote controls" are usually near "couches" and "milk" is in the "kitchen." LGX taps into this latent knowledge to turn "uninformed" wandering into "informed" searching.
  • Prompt Engineering is Critical: The authors found that how you ask the LLM matters. Providing a list of detected objects (YOLO → LLM) resulted in better navigation than providing natural language captions (BLIP → LLM). Furthermore, placing the list of objects at the beginning of the prompt helped the LLM reason more effectively.
  • GLIP's Superiority in the Wild: The paper demonstrates that GLIP is significantly more robust than previous models like CLIP or OWL-ViT when dealing with the ambiguity of natural language, specifically in its ability to differentiate between similar objects based on descriptive modifiers.

Key Results

  • State-of-the-Art Performance: On the RoboTHOR benchmark, LGX achieved a 35.0% Success Rate (SR), outperforming the previous best baseline (OWL-CoW) which sat at 27.5%.
  • Efficiency: LGX also improved the SPL (Success weighted by Path Length), meaning it didn't just find the objects—it found them using more direct, efficient paths.
  • Real-World Success: In physical robot trials, LGX achieved a 54.2% success rate in a complex two-phase navigation task, compared to just 6.9% for a robot using random exploration and 27.8% for a non-LLM baseline.

Limitations and Future Work

Despite the gains, the authors acknowledge several hurdles:

  • Memory Constraints: The current system doesn't have a long-term memory of where it has been or what it has seen in previous steps. This can sometimes lead to the robot "looping" or re-exploring the same area.
  • LLM Hallucinations: Occasionally, the LLM's common sense fails or it makes a poor association (e.g., thinking a jacket should be near a desk), leading the robot astray.
  • Small Object Detection: Very small or transparent objects (like sugar packets) remain a challenge for current vision-language models to ground accurately from a distance.
  • Future Directions: The authors suggest incorporating "history" into the LLM prompt to prevent re-exploration and exploring more advanced vision-language models like Detic to improve grounding.

Summary updated

Abstract

We present language-guided exploration (LGX), a novel algorithm for Language-Driven Zero-Shot Object Goal Navigation (L-ZSON), where an embodied agent navigates to an uniquely described target object in a previously unseen environment. Our approach makes use of large language models (LLMs) for this task by leveraging the LLM's commonsense-reasoning capabilities for making sequential navigational decisions. Simultaneously, we perform generalized target object detection using a pre-trained Vision-Language grounding model. We achieve state-of-the-art zero-shot object navigation results on RoboTHOR with a success rate (SR) improvement of over 27% over the current baseline of the OWL-ViT CLIP on Wheels (OWL CoW). Furthermore, we study the usage of LLMs for robot navigation and present an analysis of various prompting strategies affecting the model output. Finally, we showcase the benefits of our approach via real-world experiments that indicate the superior performance of LGX in detecting and navigating to visually unique objects.
BibTeX
@article{dorbala2024can,
  author = {Vishnu Sashank Dorbala and James F. Mullen Jr. and Dinesh Manocha},
  title = {{Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Based Zero-Shot Object Navigation}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
  doi = {10.1109/lra.2023.3346800},
}

Loading related content...

Figure