Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models

Xinpeng Ding, Jianhua Han, Hang Xu, Xiaodan Liang, Wei Zhang, Xiaomeng Li

Overview

Autonomous driving requires a sophisticated understanding of the environment that goes beyond simple object detection. Human drivers naturally follow a logical progression: they perceive objects, predict their future states, assess potential risks, and then plan their actions. However, most current Multi-modal Large Language Models (MLLMs) are limited to isolated tasks and often lack the 360-degree spatial awareness necessary for safe navigation.

To address these "blind spots," researchers have introduced NuInstruct, a comprehensive dataset of 91,000 multi-view video-QA pairs, and BEV-InMLLM, a model architecture that injects Bird’s-Eye-View (BEV) spatial features directly into the reasoning process.

The holistic task structure of NuInstruct, showing the progression from perception to planning across multiple camera views.

Methodology

NuInstruct: A Dataset for Holistic Reasoning

The NuInstruct dataset was created using a novel SQL-based automated pipeline. By treating driving scene metadata—such as object trajectories and distances—as a database, the authors generated complex instruction-response pairs. These pairs are structured to mirror human driving logic, covering four key stages:

  • Perception: Identifying the closest objects or specific entities.
  • Prediction: Determining the future status or motion of objects.
  • Risk Assessment: Recognizing potential threats like overtaking vehicles.
  • Planning: Formulating a safe driving plan with supporting reasons.

BEV-InMLLM Architecture

The proposed model, BEV-InMLLM, enhances a base Multi-View MLLM with a specialized BEV Injection Module. While the model processes standard perspective camera views, it also incorporates features from a pre-trained BEV extractor (e.g., BEVDet). An "Instruction-aware BEV Q-Former" ensures that only the spatial data relevant to the specific user query is utilized, providing the model with a consistent 3D coordinate system for more accurate distance and position reasoning.

Dataset Comparison

NuInstruct stands out from existing benchmarks by requiring simultaneous use of multi-view, temporal, and spatial information across a wide range of tasks. Unlike previous datasets that focus on single-view images or isolated tasks, NuInstruct demands a holistic understanding of the driving environment.

Comparison of NuInstruct with other autonomous driving datasets, highlighting its comprehensive task coverage and information richness.

Key Results and Insights

The integration of BEV features and multi-view temporal data yielded significant performance gains:

  • Performance Boost: BEV-InMLLM achieved a 9% overall improvement over state-of-the-art models like Video-LLama and MiniGPT-4.
  • Logical Tasks: Risk assessment and planning tasks saw improvements of 5% and 6%, respectively, due to better temporal and multi-view integration.
  • Spatial Accuracy: The BEV injection specifically mitigated hallucinations in "spatial-sensitive" tasks involving distance and positioning, where perspective-only models typically struggle.
  • Multi-View Necessity: Statistical analysis confirmed that answering NuInstruct questions correctly requires integrating information from multiple camera views, validating the dataset's design.

Conclusion

By bridging the gap between high-level language reasoning and low-level spatial awareness, BEV-InMLLM represents a significant step toward holistic autonomous driving understanding. While the current model does not yet incorporate traffic light data or full 3D object detection, future work aims to expand the dataset to include more edge cases and deeper integration of additional sensor modalities like LiDAR.

Source: Holistic Autonomous Driving Understanding by Bird’s-Eye-View Injected Multi-Modal Large Models (2024).

Summary updated

Abstract

The rise of multimodal large language models (MLLMs) has spurred interest in language-based driving tasks. However, existing research typically focuses on limited tasks and often omits key multi-view and temporal information which is crucial for robust autonomous driving. To bridge these gaps, we introduce NuInstruct, a novel dataset with 91K multi-view video-QA pairs across 17 subtasks, where each task demands holistic information (e.g., temporal, multi-view, and spatial), significantly elevating the challenge level. To obtain NuInstruct, we propose a novel SQL-based method to generate instruction-response pairs automatically, which is inspired by the driving logical progression of humans. We further present BEV-InMLLM, an end-to-end method for efficiently deriving instruction-aware Bird's-Eye-View (BEV) features, language-aligned for large language models. BEV-InMLLM integrates multi-view, spatial awareness, and temporal semantics to enhance MLLMs' capabilities on NuInstruct tasks. Moreover, our proposed BEV injection module is a plug-and-play method for existing MLLMs. Our experiments on NuInstruct demonstrate that BEV-InMLLM significantly outperforms existing MLLMs, e.g. around 9% improvement on various tasks. We plan to release our NuInstruct for future research development.
BibTeX
@inproceedings{ding2024holistic,
  author = {Xinpeng Ding and Jianhua Han and Hang Xu and Xiaodan Liang and Wei Zhang and Xiaomeng Li},
  title = {{Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01297},
}
Figure