Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving

Tushar Choudhary, Vikrant Dewangan, Shivam Chandhok, Shubham Priyadarshan, Anushka Jain, Arun K. Singh, Siddharth Srivastava, Krishna Murthy Jatavallabhula, K. Madhava KrishnaICRA 2023 · OpenReview

TL;DR

This paper introduces Talk2BEV, a novel system that enhances Bird's-Eye View (BEV) maps with language-based object descriptions, enabling large vision-language models (LVLMs) to perform complex reasoning and decision-making for autonomous driving (AD). They also release Talk2BEV-Bench, a new benchmark to evaluate LVLMs on AD-specific tasks.

What Problem Does This Paper Address?

Autonomous driving systems need to understand their environment comprehensively to navigate safely. Traditional AD systems often rely on specialized models for each task (e.g., object detection, trajectory prediction). This approach is rigid and struggles with unforeseen scenarios. The authors aim to bridge this gap by leveraging the power of general-purpose LVLMs, which have shown remarkable capabilities in understanding and reasoning about visual and textual information. The challenge lies in effectively integrating these powerful LVLMs with the structured BEV representations commonly used in AD, allowing them to go beyond simple object recognition to perform nuanced spatial reasoning, intent prediction, and decision-making.

Methods

Talk2BEV's core innovation is the creation of "language-enhanced BEV maps." This process involves:

  1. BEV Map Generation: Standard BEV maps are generated from multi-view images and LiDAR data using existing models like Lift-Splat-Shoot.
  2. Language Enhancement: For each object in the BEV map, the system projects it onto the corresponding camera images. Then, it uses an off-the-shelf LVLM to generate detailed textual descriptions for each object's image crop. These descriptions capture semantics, attributes, and even background context.
  3. LVLM Integration: These language-enhanced BEV maps, which combine geometric information with rich textual descriptions, are then fed as context to a powerful LVLM (like GPT-4). The LVLM can then interpret natural language queries about the scene, perform visual and spatial reasoning, and even suggest actions or predict future scenarios.

A key aspect is that Talk2BEV requires no task-specific training or fine-tuning, relying on the pre-trained capabilities of existing LVLMs. They also introduce "spatial operators" – a set of API calls that the LVLM can invoke to perform explicit spatial reasoning (e.g., calculating distances, finding nearest objects), significantly improving performance on such tasks.

Datasets

  • NuScenes dataset: Used for generating the ground-truth BEV maps and creating the Talk2BEV-Bench benchmark.
  • Talk2BEV-Bench: A newly introduced benchmark comprising 1000 human-annotated BEV scenarios from NuScenes, with over 20,000 question-answer pairs covering instance attributes, instance counting, visual reasoning, and spatial reasoning.

Key Insights and Ideas

  • LVLMs as a Unified Perception System: The paper demonstrates that LVLMs, when provided with rich, language-enhanced contextual information (like the BEV maps), can act as a single, versatile system for various AD perception tasks, moving away from the need for specialized models.
  • The Power of Language-Enhanced BEV: Simply augmenting BEV maps with textual descriptions unlocks deeper understanding. This allows LVLMs to reason about object affordances, potential interactions, and even subtle cues like reversing lights.
  • Explicit Spatial Operators are Crucial: While LVLMs are powerful, they struggle with precise spatial reasoning when left to their own devices. Providing them with an API of spatial operators significantly boosts their performance on tasks involving distances, relative positions, and object proximity.
  • BEV Map Quality vs. LVLM Quality: The study found that errors in the BEV map prediction had a relatively minor impact (around 3%) on overall performance, suggesting that the quality of the LVLM's interpretation and captioning is more critical.

Key Results

  • Quantitative Performance: Talk2BEV, especially when integrated with spatial operators, achieved significant improvements in spatial reasoning tasks, with a 58% increase in Jaccard index and a 0.09m reduction in distance error compared to relying solely on the LLM.
  • LVLM Comparison: Different LVLMs (BLIP-2, InstructBLIP-2, MiniGPT-4) showed varying strengths. InstructBLIP-2 and MiniGPT-4 generally performed better on instance attribute recognition and visual reasoning, while MiniGPT-4 excelled in instance counting.
  • Object Category Performance: Larger vehicles like trucks and construction vehicles consistently showed better performance due to their more accurate BEV segmentations. 2-wheelers were more challenging.
  • Qualitative Examples: The paper showcases impressive qualitative results, including a free-form conversation where Talk2BEV correctly identified a reversing vehicle as a hazard and advised caution, demonstrating its ability to predict future actions and provide actionable advice.

Limitations and Future Work

  • Limitations:
    • The performance on 2-wheeler categories was lower due to smaller BEV segmentation predictions.
  • While the system is zero-shot, the quality of the underlying LVLM and BEV prediction model still impacts overall performance.
  • Future Work:
    • The authors emphasize the need for further research into safety and alignment before deploying such powerful models in safety-critical AD applications.
  • The framework is designed to be flexible, allowing for easy integration of newer and more performant LVLMs as they become available.
  • The benchmark is intended to spur further research in LVLMs for AD.

Summary updated

Abstract

This work introduces Talk2BEV, a large vision-language model (LVLM) 1 interface for bird’s-eye view (BEV) maps commonly used in autonomous driving. While existing perception systems for autonomous driving scenarios have largely focused on a pre-defined (closed) set of object categories and driving scenarios, Talk2BEV eliminates the need for BEV-specific training, relying instead on well-performing pre-trained LVLMs. This enables a single system to cater to a variety of autonomous driving tasks encompassing visual and spatial reasoning, predicting the intents of traffic actors, and decision-making based on visual cues. We extensively evaluate Talk2BEV on a large number of scene understanding tasks that rely on both the ability to interpret freeform natural language queries, and in grounding these queries to the visual context embedded into the language-enhanced BEV map. To enable further research in LVLMs for autonomous driving scenarios, we develop and release Talk2BEV-Bench, a benchmark encompassing 1000 human-annotated BEV scenarios, with more than 20,000 questions and ground-truth responses from the NuScenes dataset. We encourage the reader to view the demos on our project page: https://llmbev.github.io/talk2bev/
BibTeX
@inproceedings{choudhary2023talk,
  author = {Tushar Choudhary and Vikrant Dewangan and Shivam Chandhok and Shubham Priyadarshan and Anushka Jain and Arun K. Singh and Siddharth Srivastava and Krishna Murthy Jatavallabhula and K. Madhava Krishna},
  title = {{Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=Ec15UnMj5TY},
}
Figure