TL;DR
This paper introduces CoNVOI, a novel robot navigation system that leverages Vision Language Models (VLMs) to understand environmental context and generate human-like navigation behaviors. By combining VLM's reasoning with a unique visual marking system, CoNVOI achieves context-aware navigation in complex indoor and outdoor environments without requiring task-specific training.
What Problem Does This Paper Address?
Navigating robots in real-world environments, especially those with dynamic elements like humans or varying terrain, requires more than just obstacle avoidance. Robots need to exhibit "socially acceptable" behaviors (e.g., not cutting through groups of people) and adapt to different contexts (e.g., using crosswalks outdoors, staying on pavement). While existing methods excel at specific tasks like obstacle avoidance or terrain classification, they often lack the generalization and nuanced understanding needed for these complex, human-like behaviors. Furthermore, relying on VLMs for direct control is challenging due to their computational demands and slow response times. This paper addresses the gap of enabling robots to perform context-aware, human-like navigation in diverse environments by intelligently integrating VLMs into the navigation pipeline.
Methods
CoNVOI employs a two-pronged approach using VLMs:
- Context Identification: A lightweight VLM (like CLIP) is used to interpret the robot's surroundings and classify the current context (e.g., "indoor corridor," "outdoor terrain," "crosswalk").
- Context-Aware Behavior Generation: A large VLM (like GPT-4v or Gemini) is then prompted with the identified context and a corresponding behavioral text prompt (e.g., "stay on the crosswalk").
- Multi-modal Visual Marking: To guide the VLM's attention and improve its understanding of spatial relationships, CoNVOI introduces a novel visual marking system. This system overlays numbered markers on navigable, obstacle-free regions in the RGB image, correlating them with an occupancy grid map. This helps the VLM focus on relevant areas and understand their spatial context.
- Reference Path Generation: The VLM, using the marked image and text prompt, generates a reference path by selecting a sequence of these markers. This path encodes the desired context-specific behavior.
- Extrapolation and Re-querying: To mitigate the high inference time of large VLMs, CoNVOI implements a strategy to extrapolate the generated reference path when the environment context remains unchanged. The system only re-queries the VLM when a context change is detected or when extrapolation is no longer reliable.
- Motion Planning: The generated reference path is then fed to a standard motion planner (specifically, the dynamic window approach) to control the robot's velocity commands, ensuring smooth and real-time navigation.
Datasets
Not explicitly discussed in the paper. The evaluation uses real-world indoor and outdoor scenarios with robots (Turtlebot 2 and Boston Dynamics Spot) and human teleoperated ground truth.
Key Insights and Ideas
- VLMs as "Contextual Navigators": The core insight is that VLMs, when properly prompted and guided, can act as sophisticated "navigators" by understanding environmental context and generating high-level behavioral guidance, rather than just performing low-level control.
- Visual Marking for VLM Grounding: The multi-modal visual marking is crucial. It transforms abstract VLM capabilities into concrete, actionable guidance by grounding visual information with spatial context and highlighting navigable areas, leading to more accurate VLM outputs.
- Zero-Shot Social and Contextual Navigation: CoNVOI demonstrates that VLMs can achieve complex, human-like behaviors (social compliance, context-specific actions) in a zero-shot manner, without requiring extensive task-specific training or fine-tuning.
- Balancing VLM Power with Real-time Constraints: The extrapolation mechanism is key to making VLM-based navigation practical. It intelligently reduces the reliance on slow VLM queries, enabling real-time performance.
Key Results
- Human-like Trajectories: CoNVOI's trajectories closely match human teleoperated ground truth, achieving significantly lower Fréchet distances (9.7-58.2% closer) compared to baseline methods.
- Reduced Path Errors: The proposed method leads to lower path errors (up to 88.13% lower) and a significantly lower percentage of unacceptable paths (up to 86.09% fewer).
- Improved VLM Performance: The ablation studies show that both multi-modal visual marking (MMVM) and context-based prompting (CbP) significantly improve VLM performance in generating human-like reference paths, leading to lower reference path errors and higher cosine similarity with ground truth.
- Efficient Context Handling: CoNVOI can handle new scenarios with simple text prompts without reformulation, as demonstrated by adding a detour scenario.
- Extrapolation Benefits: Extrapolation reduces VLM requeries by approximately 50% in static environments.
Limitations and Future Work
- VLM Inference Time and Network Latency: The reliance on remote VLMs introduces processing time and network latency, limiting the robot's maximum velocity. This also makes it challenging for highly dynamic environments with crowds.
- VLM Hallucinations: VLMs can sometimes "hallucinate" patterns in their outputs, especially with repetitive inputs. The paper addresses this by adding phrases to the prompt to treat each query as a new task.
- Sensitivity to Vague Prompts: The system can be sensitive to vague text prompts, highlighting the need for precise language.
- Lack of Global Context: CoNVOI currently relies on local environmental information and does not use global context (e.g., top-down maps) which could be beneficial for long-range navigation or complex decision-making (e.g., choosing where to cross a road).
Future work includes addressing these limitations by developing faster, locally executable VLMs, improving prompt engineering, and incorporating global environmental information for long-range navigation.
Summary updated