Overview
Autonomous driving requires a sophisticated understanding of the environment that goes beyond simple object detection. Human drivers naturally follow a logical progression: they perceive objects, predict their future states, assess potential risks, and then plan their actions. However, most current Multi-modal Large Language Models (MLLMs) are limited to isolated tasks and often lack the 360-degree spatial awareness necessary for safe navigation.
To address these "blind spots," researchers have introduced NuInstruct, a comprehensive dataset of 91,000 multi-view video-QA pairs, and BEV-InMLLM, a model architecture that injects Bird’s-Eye-View (BEV) spatial features directly into the reasoning process.
Methodology
NuInstruct: A Dataset for Holistic Reasoning
The NuInstruct dataset was created using a novel SQL-based automated pipeline. By treating driving scene metadata—such as object trajectories and distances—as a database, the authors generated complex instruction-response pairs. These pairs are structured to mirror human driving logic, covering four key stages:
- Perception: Identifying the closest objects or specific entities.
- Prediction: Determining the future status or motion of objects.
- Risk Assessment: Recognizing potential threats like overtaking vehicles.
- Planning: Formulating a safe driving plan with supporting reasons.
BEV-InMLLM Architecture
The proposed model, BEV-InMLLM, enhances a base Multi-View MLLM with a specialized BEV Injection Module. While the model processes standard perspective camera views, it also incorporates features from a pre-trained BEV extractor (e.g., BEVDet). An "Instruction-aware BEV Q-Former" ensures that only the spatial data relevant to the specific user query is utilized, providing the model with a consistent 3D coordinate system for more accurate distance and position reasoning.
Dataset Comparison
NuInstruct stands out from existing benchmarks by requiring simultaneous use of multi-view, temporal, and spatial information across a wide range of tasks. Unlike previous datasets that focus on single-view images or isolated tasks, NuInstruct demands a holistic understanding of the driving environment.
Key Results and Insights
The integration of BEV features and multi-view temporal data yielded significant performance gains:
- Performance Boost: BEV-InMLLM achieved a 9% overall improvement over state-of-the-art models like Video-LLama and MiniGPT-4.
- Logical Tasks: Risk assessment and planning tasks saw improvements of 5% and 6%, respectively, due to better temporal and multi-view integration.
- Spatial Accuracy: The BEV injection specifically mitigated hallucinations in "spatial-sensitive" tasks involving distance and positioning, where perspective-only models typically struggle.
- Multi-View Necessity: Statistical analysis confirmed that answering NuInstruct questions correctly requires integrating information from multiple camera views, validating the dataset's design.
Conclusion
By bridging the gap between high-level language reasoning and low-level spatial awareness, BEV-InMLLM represents a significant step toward holistic autonomous driving understanding. While the current model does not yet incorporate traffic light data or full 3D object detection, future work aims to expand the dataset to include more edge cases and deeper integration of additional sensor modalities like LiDAR.
Source: Holistic Autonomous Driving Understanding by Bird’s-Eye-View Injected Multi-Modal Large Models (2024).
Summary updated