PAVE: An End-to-End Dataset for Production Autonomous Vehicle Evaluation

Xiang-Yang Li, Chen Wang, Yumao Liu, Dengbo He, Jiahao Zhang, Ke MaarXiv 2025 · DOI · arXiv

Introduction

The Production Autonomous Vehicle Evaluation (PAVE) dataset represents a significant shift in autonomous driving research. While established datasets like KITTI, nuScenes, and Waymo have been instrumental in advancing perception systems, they often lack the specific context needed to evaluate how production-grade autonomous vehicles (AVs) actually make decisions. Most existing data is recorded during human operation or in unidentified driving modes, leaving a "safety gap" in our understanding of "black-box" systems like Tesla FSD or Huawei ADS.

PAVE addresses this by providing over 140 hours of real-world data collected directly from production AVs operating in their native autonomous modes. By including explicit "human vs. autonomous" labels, PAVE enables researchers to analyze the behavioral safety and decision-making logic of commercial systems in complex, "long-tail" scenarios.

Dataset Characteristics and Comparison

The PAVE dataset is distinguished by its scale, diversity, and precision. It encompasses 32,727 key frames collected from five different anonymized production vehicle brands across seven major cities in China and the USA.

Comparison of PAVE with existing autonomous driving datasets, highlighting its unique labeled driving modes and high localization accuracy.

As shown in the comparison above, PAVE offers a unique combination of labeled driving modes and industry-leading localization accuracy (0.008m), achieved through Real-Time Kinematic (RTK) corrections. Unlike other end-to-end datasets that blend human and autonomous data, PAVE’s explicit labeling allows for direct comparative studies of driving behavior.

Data Collection and Annotation

The researchers utilized a standardized sensor platform integrated into multiple production models. Each vehicle features:

  • Sensor Suite: Four high-resolution RGB cameras providing a wide field of view, paired with a high-precision GNSS/IMU module.
  • Temporal Segmentation: Data is organized into 11-second segments, centered on a "key frame" that provides 6 seconds of history and 5 seconds of future trajectory.
  • Rich Metadata: Beyond 2D bounding boxes, the dataset includes manual annotations for scenario-level attributes such as weather, lighting, road surface, and the density of Vulnerable Road Users (VRUs).
Visual examples of the diverse driving scenarios captured in the PAVE dataset, including various weather conditions and traffic densities.

To ensure privacy, an automated pipeline applies Gaussian blurring to faces and license plates, complying with international data protection regulations.

Key Insights: The Predictability Gap

Analysis of the PAVE dataset has revealed a striking "predictability gap" between human and autonomous drivers. Baseline motion planning models consistently find human-driven trajectories easier to predict, with an Average Displacement Error (ADE) of 1.20m compared to 1.76m for AVs. This suggests that current production AV behavior is less "natural" or consistent than human driving, particularly at higher speeds on highways.

Furthermore, the dataset's identification of specific vehicle models enables "inverse modeling," allowing researchers to reverse-engineer and study the distinct driving logics employed by different commercial brands.

Results and Future Outlook

Initial benchmarks using YOLOv5 for detection and E2EDriver for motion planning established a baseline mean Average Precision (mAP) of 0.310 and an ADE of 1.47m across the full dataset. While the model performs well on standard passenger cars, it faces challenges with rarer targets like motorcycles and extreme weather conditions.

The PAVE project is envisioned as a "living" dataset, with 10 hours of new data added weekly. Future work will focus on expanding the scale of "long-tail" events and developing a dedicated safety evaluation framework to quantify driving comfort and risk beyond simple displacement metrics.

Summary updated

BibTeX
@misc{li2025pave,
  author = {Xiang-Yang Li and Chen Wang and Yumao Liu and Dengbo He and Jiahao Zhang and Ke Ma},
  title = {{PAVE: An End-to-End Dataset for Production Autonomous Vehicle Evaluation}},
  year = {2025},
  eprint = {2511.14185},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.14185},
}
Figure