OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning

Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, Jose M. Alvarez

TL;DR

OmniDrive introduces a novel vision-language dataset for autonomous driving that leverages counterfactual ("what-if") reasoning to generate high-quality 3D question-answering data. By evaluating both safe and unsafe simulated trajectories, it provides dense supervision for driving agents, demonstrating that adapting 2D Vision-Language Models (VLMs) to 3D spaces outperforms traditional 3D perception-heavy architectures.

What Problem Does This Paper Address?

While Vision-Language Models (VLMs) excel at 2D reasoning, real-world autonomous driving demands full 3D spatial and geometric understanding. Existing driving datasets mostly provide sparse supervision—they only show the "expert" (safe) trajectory. They fail to capture why a driver made a choice or what would happen if they chose an alternative path.

Consequently, models trained on these datasets often overfit to expert paths without developing true causal reasoning, leading to poor generalization in complex or novel scenarios. The authors address this gap by introducing counterfactual reasoning into the data generation process, forcing the model to evaluate potential scenarios and their outcomes.

Figure 1. OmniDrive is a holistic vision-language dataset for autonomous driving, utilizing counterfactual reasoning to generate high-quality QA data from simulated and actual trajectories. We explore two baseline models: Omni-Q, which designs vision-language models (VLMs) from a 3D perception standpoint, and Omni-L, which builds from VLMs to enhance 3D integration.

Methods

The authors propose a human-in-the-loop data generation pipeline and two distinct model architectures to evaluate different learning paradigms.

Data Generation Pipeline

  1. Key-Frame Selection: To reduce data redundancy, the pipeline uses CLIP embeddings and K-means clustering to select the top 20% most semantically diverse frames and vehicle dynamics from the nuScenes dataset.
  2. Counterfactual Simulation: For each scene, the system simulates alternative driving behaviors (e.g., turning left when the expert went straight, accelerating, or stopping).
  3. Rule-Based Checklist: A checklist utilizes 3D object detection and map topology (from nuScenes and OpenLane-v2) to automatically determine if a simulated trajectory violates traffic rules, causes a collision, or leaves the drivable area.
  4. GPT-4 Q&A Generation: The simulated trajectories, checklist results, and expert trajectories are fed into GPT-4 to generate coherent, multi-turn Q&A data covering scene description, attention allocation, counterfactual reasoning, and planning.
Data Generation Pipeline

Model Frameworks

The authors explore two advanced OmniDrive-Agent frameworks:

  • Omni-L (VLM-Centric): Built on top of LLaVA, this model flattens multi-view image features and feeds them directly into an LLM. It introduces 3D position encodings to each image patch to help the model distinguish different viewpoints.
  • Omni-Q (Perception-Centric): Built on the 3D bird's-eye-view (BEV) architecture of StreamPETR, this model incorporates a Q-Former to bridge traditional query-based 3D perception tasks with the LLM token space.
Omni-L vs Omni-Q Architectures

Datasets

  • OmniDrive (New): A synthetic, counterfactual-centric 3D driving Q&A dataset built on top of nuScenes and OpenLane-v2 annotations.
  • DriveLM: Used as an external benchmark to evaluate the generalization capability of the pre-trained OmniDrive models.
  • nuScenes: Used for evaluating open-loop planning performance (L2 error, collision rate, and road boundary intersection rate).

Key Insights and Ideas

  • The Power of "What-Not-To-Do": Teaching an autonomous agent what not to do via counterfactual reasoning provides a significantly denser and more effective supervision signal than simply imitating safe expert trajectories.
  • VLM-First Trumps Perception-First: A major finding is that migrating 2D VLMs to 3D spaces (Omni-L) is a more straightforward and effective approach than trying to integrate traditional 3D perception stacks into VLMs (Omni-Q). Omni-L retained stronger foundational language and reasoning skills, which translated to better overall planning.
  • The Ego-Status Trap: In open-loop planning benchmarks, models often overfit to the vehicle's current speed and status ("ego status"). Training models with counterfactual Q&A data without ego status effectively mitigates this bias, forcing the model to rely on visual reasoning rather than statistical shortcuts.

Key Results

  • DriveLM Benchmark: Pre-training Omni-L on the OmniDrive dataset improved its total score on the DriveLM benchmark from 0.53 to 0.56. When combined with LLaVA665k pre-training, the score reached 0.58.
  • nuScenes Open-Loop Planning: When evaluated without relying on ego status, the Omni-L++ model significantly outperformed the baseline, reducing the collision rate from 3.29% to 1.90% and the road boundary intersection rate from 6.16% to 3.29%.
  • Counterfactual Reasoning Performance: In ablation studies, Omni-L achieved a higher precision and recall across safety, red-light, collision, and drivable area tasks compared to Omni-Q and BEV-MLP baselines.

Limitations and Future Work

The primary limitation acknowledged by the authors is that their counterfactual simulations are currently static and open-loop. When the ego vehicle simulates an alternative action, the pipeline does not yet model how other dynamic agents (pedestrians, other vehicles) would react to that action.

As a future direction, the authors plan to transition from static log-replays to closed-loop simulators to capture multi-agent interactions and further enhance the real-world effectiveness of the dataset.

Summary updated

Abstract

The advances in vision-language models (VLMs) have led to a growing interest in autonomous driving to leverage their strong reasoning capabilities. However, extending these capabilities from 2D to full 3D understanding is crucial for real-world applications. To address this challenge, we propose OmniDrive, a holistic vision-language dataset that aligns agent models with 3D driving tasks through counterfactual reasoning. This approach enhances decision-making by evaluating potential scenarios and their outcomes, similar to human drivers considering alternative actions. Our counterfactual-based synthetic data annotation process generates large-scale, high-quality datasets, providing denser supervision signals that bridge planning trajectories and language-based reasoning. Futher, we explore two advanced OmniDrive-Agent frameworks, namely Omni-L and Omni-Q, to assess the importance of vision-language alignment versus 3D perception, revealing critical insights into designing effective LLM-agents. Significant improvements on the DriveLM Q\&A benchmark and nuScenes open-loop planning demonstrate the effectiveness of our dataset and methods.
BibTeX
@inproceedings{wang2025omnidrive,
  author = {Shihao Wang and Zhiding Yu and Xiaohui Jiang and Shiyi Lan and Min Shi and Nadine Chang and Jan Kautz and Ying Li and Jose M. Alvarez},
  title = {{OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=5E5d9hVcmL},
}
Figure