OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning

Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, José M. Álvarez

TL;DR

This paper introduces OmniDrive, a novel vision-language dataset for autonomous driving that leverages counterfactual reasoning to generate rich, 3D-aware question-answering data. They also propose two VLM frameworks, Omni-L and Omni-Q, demonstrating that vision-language alignment is crucial for effective 3D driving agents, outperforming models that solely rely on 3D perception.

What Problem Does This Paper Address?

Autonomous driving systems are increasingly leveraging the power of Vision-Language Models (VLMs) for their reasoning capabilities. However, a significant challenge remains in extending these models from 2D understanding to the full 3D spatial comprehension required for real-world driving. Existing datasets often provide sparse supervision, focusing on safe trajectories rather than the complex decision-making processes and underlying reasoning that human drivers employ. This gap makes it difficult to train end-to-end driving models that can truly understand and reason about 3D driving scenarios, especially when considering alternative actions and their potential outcomes.

Methods

The core of OmniDrive's methodology lies in its innovative data generation pipeline, which combines counterfactual reasoning with VLMs. Here's a breakdown:

  • Counterfactual Reasoning for Data Generation: Instead of relying solely on expert trajectories, OmniDrive simulates various driving behaviors (e.g., accelerating, turning) and uses these simulated trajectories to generate counterfactual scenarios. This allows for the creation of denser supervision signals that explicitly link planning trajectories with language-based reasoning.
  • GPT-4 for QA Generation: The paper leverages GPT-4's capabilities to process complex 3D scene information and generate high-quality question-answering (QA) pairs. This is achieved by first creating a structured, simplified 3D scene representation and then designing specific prompts for GPT-4.
  • Rule-Based Checklist and Human-in-the-Loop: To ensure data quality and interpretability, a rule-based checklist is used to assess the consequences of potential trajectories. A human-in-the-loop approach is also employed in designing these checklists and prompts, ensuring comprehensive coverage of various driving scenarios.
  • Omni-L and Omni-Q Frameworks: Two VLM frameworks are proposed to assess different design philosophies for autonomous driving agents:
  • Omni-L: This framework extends existing 2D VLMs (like LLaVA) by incorporating 3D positional encoding and aligning multi-view image features with language. It prioritizes vision-language alignment.
  • Omni-Q: This framework builds upon a Bird's-Eye View (BEV) architecture and incorporates a Q-Former design, aiming to integrate LLMs with traditional 3D autonomous driving perception tasks. It focuses more on 3D perception.

Datasets

  • OmniDrive (New Dataset): The paper introduces OmniDrive, a holistic vision-language dataset specifically designed for autonomous driving with counterfactual reasoning. It's generated using simulated and actual trajectories, enriched with counterfactual reasoning to provide dense supervision signals for planning and language-based reasoning.
  • nuScenes: Used as the base dataset for generating OmniDrive's synthetic data.
  • DriveLM: Used for evaluating the performance of models trained on OmniDrive, particularly for end-to-end autonomous driving and QA tasks.

Key Insights and Ideas

  • Counterfactual Reasoning is Key for 3D Driving Understanding: The paper strongly suggests that simulating "what-if" scenarios is crucial for developing VLMs that can truly grasp the complexities of 3D driving, mirroring how human drivers make decisions.
  • Vision-Language Alignment Trumps Pure 3D Perception for VLMs: The experimental results highlight that models prioritizing vision-language alignment (Omni-L) tend to perform better in 3D driving tasks compared to those heavily reliant on traditional 3D perception (Omni-Q), especially when not explicitly leveraging ego status.
  • Structured Scene Representation Simplifies VLM Input: Directly feeding raw images and complex 3D data to VLMs can lead to overlooked details. By creating a simplified, structured scene representation, the paper shows how to make 3D information more digestible for LLMs.
  • Human-in-the-Loop is Essential for High-Quality Data: Despite the power of LLMs like GPT-4, human oversight in designing checklists and prompts is vital for ensuring the reliability and interpretability of the generated QA data.

Key Results

  • OmniDrive Improves Performance: Models pre-trained on OmniDrive showed significant improvements on the DriveLM Q&A benchmark and nuScenes open-loop planning tasks.
  • Omni-L Outperforms Omni-Q: Omni-L, which focuses on vision-language alignment, generally performed better on counterfactual reasoning tasks (e.g., 72.1% Precision and 58.0% Recall in safety tasks) compared to Omni-Q.
  • DriveLM Benchmark Gains: Training with OmniDrive boosted performance on the DriveLM benchmark by 3% compared to training solely on DriveLM. Further improvements were seen when combined with LLaVA665k pre-training.
  • Reduced Collision Rates: Training with OmniDrive Q&A data, even without ego status, effectively reduced collision rates from 3.22% to 1.90% and intersection rates from 3.90% to 3.29%.

Limitations and Future Work

  • Simulation Limitations: The current simulation of counterfactual outcomes does not account for the reactions of other agents in the scene.
  • Future Work: The authors plan to integrate closed-loop planning simulators to enhance the effectiveness of their approach, especially for scenarios involving interactions with other agents. They also suggest further research into aligning traditional 3D perception stacks with language spaces to improve performance.

Summary updated

Abstract

The advances in vision-language models (VLMs) have led to a growing interest in autonomous driving to leverage their strong reasoning capabilities. However, extending these capabilities from 2D to full 3D understanding is crucial for real-world applications. To address this challenge, we propose OmniDrive, a holistic vision-language dataset that aligns agent models with 3D driving tasks through counterfactual reasoning. This approach enhances decision-making by evaluating potential scenarios and their outcomes, similar to human drivers considering alternative actions. Our counterfactual-based synthetic data annotation process generates large-scale, high-quality datasets, providing denser supervision signals that bridge planning trajectories and language-based reasoning. Futher, we explore two advanced OmniDrive-Agent frameworks, namely Omni-L and Omni-Q, to assess the importance of vision-language alignment versus 3D perception, revealing critical insights into designing effective LLM-agents. Significant improvements on the DriveLM Q\&A benchmark and nuScenes open-loop planning demonstrate the effectiveness of our dataset and methods.
BibTeX
@inproceedings{wang2025omnidrive,
  author = {Shihao Wang and Zhiding Yu and Xiaohui Jiang and Shiyi Lan and Min Shi and Nadine Chang and Jan Kautz and Ying Li and José M. Álvarez},
  title = {{OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=5E5d9hVcmL},
}
Figure