An interactive enhanced driving dataset for autonomous driving

Haojie Feng, Peizhi Zhang, Mengjie Tian, Xinrui Zhang, Zhuoren Li, Junpeng Huang, Xiurong Wang, Junfan Zhu, Jianzhou Wang, Dongxiao Yin, Lu Xiong

Abstract

The evolution of autonomous driving towards full automation demands robust interactive capabilities; however, the development of Vision-Language-Action (VLA) models is constrained by the sparsity of interactive scenarios and inadequate multimodal alignment in existing data. To this end, this paper proposes the Interactive Enhanced Driving Dataset (IEDD). We develop a scalable pipeline to mine million-level interactive segments from naturalistic driving data based on interactive trajectories, and design metrics to quantify the interaction processes. Furthermore, the IEDD-VQA dataset is constructed by generating synthetic Bird's Eye View (BEV) videos where semantic actions are strictly aligned with structured language. Benchmark results evaluating ten mainstream Vision Language Models (VLMs) are provided to demonstrate the dataset's reuse value in assessing and fine-tuning the reasoning capabilities of autonomous driving models.

Introduction

Current autonomous driving systems excel at routine tasks like lane-keeping or simple car-following. However, achieving full autonomy requires mastering the "long-tail" of complex human interactions, such as negotiating unprotected left turns, merging into dense traffic, or yielding at crosswalks.

Examples of complex merging and crossing interactions that are often underrepresented in standard datasets.

The primary bottleneck is data. Existing datasets are dominated by non-interactive segments and are often unimodal (e.g., only video or only coordinates). To develop models that reason like humans—the Vision-Language-Action (VLA) paradigm—we need data that bridges visual scenes with logical "why" and "how" descriptions. Manually annotating millions of hours of video is prohibitively expensive, leading to a scarcity of high-quality, multi-modal interaction data.

Methodology

To address these challenges, the researchers developed a scalable, three-stage pipeline to transform raw trajectory data into a rich VLA dataset.

The comprehensive pipeline for IEDD, spanning interaction mining, physics-aware quantification, and multimodal synthesis.

1. Interaction Mining

Instead of collecting new data, the authors "upcycled" five massive public datasets: Waymo Open Motion, nuPlan, Lyft Level 5, INTERACTION, and SIND. Spatio-temporal filters were used to identify moments where vehicles influenced each other (merging, crossing, etc.), extracting over 7.3 million segments.

2. Physics-Aware Quantification

Beyond simple labeling, the system quantifies interactions using a new "intensity-efficiency" metric based on stochastic processes. This calculates how "intense" a conflict was (e.g., Time-to-Collision) and the efficiency of the driver's response.

3. Multi-modal Synthesis (IEDD-VQA)

For compatibility with Vision Language Models (VLMs), trajectories are rendered into Bird's Eye View (BEV) videos. A rule-based template system generates strictly aligned Question-Answering (QA) pairs, ensuring language labels perfectly match the underlying physics without the hallucinations common in purely AI-generated annotations.

Key Insights and Results

The resulting IEDD dataset is significantly larger and more balanced than predecessors. While the Waymo dataset is 86% simple car-following, IEDD maintains a high proportion of high-risk scenarios like crossing and head-on interactions.

VLM Performance and Reasoning

  • Open-Source Success: In zero-shot testing, open-source models like Llama-4-Maverick and Qwen2.5-VL-7B outperformed closed-source models like GPT-4o in driving scene understanding, suggesting specialized spatial reasoning is achievable with medium-sized models.
  • The Advantage of BEV: Bird’s Eye View was chosen over standard camera feeds because it removes occlusions and provides a clean representation of road topology, which is essential for learning the physics of interactions.
  • Chain-of-Thought (CoT) Prompting: VLMs often struggle with physical value estimation. However, CoT prompting—forcing step-by-step reasoning—drastically improved numerical accuracy.

Impact of Fine-tuning

Fine-tuning a model on IEDD-VQA using LoRA (Low-Rank Adaptation) led to a 78.7% improvement in overall performance. Most notably, the Mean Absolute Error (MAE) for physical parameter estimation dropped from 1855.55 to 0.30, proving that models can learn to "see" physics when provided with high-quality, aligned data.

Limitations and Future Work

A significant challenge identified is catastrophic forgetting. While fine-tuning made models domain experts in driving, their performance on general counterfactual reasoning (predicting alternative outcomes) decreased. Future work will focus on regularization strategies to maintain general intelligence during specialized training. Additionally, the authors plan to expand the dataset to include pedestrians and cyclists.

Summary updated

Abstract

The evolution of autonomous driving towards full automation demands robust interactive capabilities; however, the development of Vision-Language-Action (VLA) models is constrained by the sparsity of interactive scenarios and inadequate multimodal alignment in existing data. To this end, this paper proposes the Interactive Enhanced Driving Dataset (IEDD). We develop a scalable pipeline to mine million-level interactive segments from naturalistic driving data based on interactive trajectories, and design metrics to quantify the interaction processes. Furthermore, the IEDD-VQA dataset is constructed by generating synthetic Bird's Eye View (BEV) videos where semantic actions are strictly aligned with structured language. Benchmark results evaluating ten mainstream Vision Language Models (VLMs) are provided to demonstrate the dataset's reuse value in assessing and fine-tuning the reasoning capabilities of autonomous driving models.
BibTeX
@misc{feng2026interactive,
  author = {Haojie Feng and Peizhi Zhang and Mengjie Tian and Xinrui Zhang and Zhuoren Li and Junpeng Huang and Xiurong Wang and Junfan Zhu and Jianzhou Wang and Dongxiao Yin and Lu Xiong},
  title = {{An interactive enhanced driving dataset for autonomous driving}},
  year = {2026},
  eprint = {2602.20575},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.20575},
}
Figure