CoNVOI: Context-aware Navigation using Vision Language Models in Outdoor and Indoor Environments

Adarsh Jagan Sathyamoorthy, Kasun Weerakoon, Mohamed Elnoor, Anuj Zore, Brian Ichter, Fei Xia, Jie Tan, Wenhao Yu, Dinesh Manocha

TL;DR

This paper introduces CoNVOI, a novel robot navigation system that leverages Vision Language Models (VLMs) to understand environmental context and generate human-like navigation behaviors. By combining VLM's reasoning with a unique visual marking system, CoNVOI achieves context-aware navigation in complex indoor and outdoor environments without requiring task-specific training.

What Problem Does This Paper Address?

Navigating robots in real-world environments, especially those with dynamic elements like humans or varying terrain, requires more than just obstacle avoidance. Robots need to exhibit "socially acceptable" behaviors (e.g., not cutting through groups of people) and adapt to different contexts (e.g., using crosswalks outdoors, staying on pavement). While existing methods excel at specific tasks like obstacle avoidance or terrain classification, they often lack the generalization and nuanced understanding needed for these complex, human-like behaviors. Furthermore, relying on VLMs for direct control is challenging due to their computational demands and slow response times. This paper addresses the gap of enabling robots to perform context-aware, human-like navigation in diverse environments by intelligently integrating VLMs into the navigation pipeline.

Methods

CoNVOI employs a two-pronged approach using VLMs:

  • Context Identification: A lightweight VLM (like CLIP) is used to interpret the robot's surroundings and classify the current context (e.g., "indoor corridor," "outdoor terrain," "crosswalk").
  • Context-Aware Behavior Generation: A large VLM (like GPT-4v or Gemini) is then prompted with the identified context and a corresponding behavioral text prompt (e.g., "stay on the crosswalk").
  • Multi-modal Visual Marking: To guide the VLM's attention and improve its understanding of spatial relationships, CoNVOI introduces a novel visual marking system. This system overlays numbered markers on navigable, obstacle-free regions in the RGB image, correlating them with an occupancy grid map. This helps the VLM focus on relevant areas and understand their spatial context.
  • Reference Path Generation: The VLM, using the marked image and text prompt, generates a reference path by selecting a sequence of these markers. This path encodes the desired context-specific behavior.
  • Extrapolation and Re-querying: To mitigate the high inference time of large VLMs, CoNVOI implements a strategy to extrapolate the generated reference path when the environment context remains unchanged. The system only re-queries the VLM when a context change is detected or when extrapolation is no longer reliable.
  • Motion Planning: The generated reference path is then fed to a standard motion planner (specifically, the dynamic window approach) to control the robot's velocity commands, ensuring smooth and real-time navigation.

Datasets

Not explicitly discussed in the paper. The evaluation uses real-world indoor and outdoor scenarios with robots (Turtlebot 2 and Boston Dynamics Spot) and human teleoperated ground truth.

Key Insights and Ideas

  • VLMs as "Contextual Navigators": The core insight is that VLMs, when properly prompted and guided, can act as sophisticated "navigators" by understanding environmental context and generating high-level behavioral guidance, rather than just performing low-level control.
  • Visual Marking for VLM Grounding: The multi-modal visual marking is crucial. It transforms abstract VLM capabilities into concrete, actionable guidance by grounding visual information with spatial context and highlighting navigable areas, leading to more accurate VLM outputs.
  • Zero-Shot Social and Contextual Navigation: CoNVOI demonstrates that VLMs can achieve complex, human-like behaviors (social compliance, context-specific actions) in a zero-shot manner, without requiring extensive task-specific training or fine-tuning.
  • Balancing VLM Power with Real-time Constraints: The extrapolation mechanism is key to making VLM-based navigation practical. It intelligently reduces the reliance on slow VLM queries, enabling real-time performance.

Key Results

  • Human-like Trajectories: CoNVOI's trajectories closely match human teleoperated ground truth, achieving significantly lower Fréchet distances (9.7-58.2% closer) compared to baseline methods.
  • Reduced Path Errors: The proposed method leads to lower path errors (up to 88.13% lower) and a significantly lower percentage of unacceptable paths (up to 86.09% fewer).
  • Improved VLM Performance: The ablation studies show that both multi-modal visual marking (MMVM) and context-based prompting (CbP) significantly improve VLM performance in generating human-like reference paths, leading to lower reference path errors and higher cosine similarity with ground truth.
  • Efficient Context Handling: CoNVOI can handle new scenarios with simple text prompts without reformulation, as demonstrated by adding a detour scenario.
  • Extrapolation Benefits: Extrapolation reduces VLM requeries by approximately 50% in static environments.

Limitations and Future Work

  • VLM Inference Time and Network Latency: The reliance on remote VLMs introduces processing time and network latency, limiting the robot's maximum velocity. This also makes it challenging for highly dynamic environments with crowds.
  • VLM Hallucinations: VLMs can sometimes "hallucinate" patterns in their outputs, especially with repetitive inputs. The paper addresses this by adding phrases to the prompt to treat each query as a new task.
  • Sensitivity to Vague Prompts: The system can be sensitive to vague text prompts, highlighting the need for precise language.
  • Lack of Global Context: CoNVOI currently relies on local environmental information and does not use global context (e.g., top-down maps) which could be beneficial for long-range navigation or complex decision-making (e.g., choosing where to cross a road).

Future work includes addressing these limitations by developing faster, locally executable VLMs, improving prompt engineering, and incorporating global environmental information for long-range navigation.

Summary updated

Abstract

We present ConVOI, a novel method for autonomous robot navigation in real-world indoor and outdoor environments using Vision Language Models (VLMs). We employ VLMs in two ways: first, we leverage their zero-shot image classification capability to identify the context or scenario (e.g., indoor corridor, outdoor terrain, crosswalk, etc) of the robot's surroundings, and formulate context-based navigation behaviors as simple text prompts (e.g. ``stay on the pavement"). Second, we utilize their state-of-the-art semantic understanding and logical reasoning capabilities to compute a suitable trajectory given the identified context. To this end, we propose a novel multi-modal visual marking approach to annotate the obstacle-free regions in the RGB image used as input to the VLM with numbers, by correlating it with a local occupancy map of the environment. The marked numbers ground image locations in the real-world, direct the VLM's attention solely to navigable locations, and elucidate the spatial relationships between them and terrains depicted in the image to the VLM. Next, we query the VLM to select numbers on the marked image that satisfy the context-based behavior text prompt, and construct a reference path using the selected numbers. Finally, we propose a method to extrapolate the reference trajectory when the robot's environmental context has not changed to prevent unnecessary VLM queries. We use the reference trajectory to guide a motion planner, and demonstrate that it leads to human-like behaviors (e.g. not cutting through a group of people, using crosswalks, etc.) in various real-world indoor and outdoor scenarios.
BibTeX
@inproceedings{sathyamoorthy2024convoi,
  author = {Adarsh Jagan Sathyamoorthy and Kasun Weerakoon and Mohamed Elnoor and Anuj Zore and Brian Ichter and Fei Xia and Jie Tan and Wenhao Yu and Dinesh Manocha},
  title = {{CoNVOI: Context-aware Navigation using Vision Language Models in Outdoor and Indoor Environments}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
}
Figure