Towards Vision Zero: The Accid3nD Dataset

Walter Zimmer, Ross Greer, Daniel Lehmberg, Marc Pavel, Holger Caesar, Xingcheng Zhou, Ahmed Alaaeldin Ghita, Mohan M. Trivedi, Rui Song, Hu Cao, Akshay Gopalkrishnan, Alois C. Knoll

Introduction

The pursuit of "Vision Zero"—a future with zero traffic fatalities—requires autonomous driving (AD) systems to handle not just everyday driving, but the "long-tail" of rare, high-risk events. Most existing AD datasets focus on nominal driving, leaving a critical gap in real-world accident data. Synthetic datasets attempt to fill this gap but suffer from a "sim-to-real" domain gap, while 2D CCTV datasets lack the depth information needed for precise 3D perception.

The TUM Traffic Accid3nD dataset addresses these limitations. It is the first public, large-scale collection of real-world highway accidents captured with 3D annotations from roadside infrastructure, providing a "digital twin" of traffic to study hazardous incidents as they occur.

The Accid3nD Dataset

The dataset comprises 111,945 labeled frames and over 2.6 million 3D bounding boxes. Unlike ego-vehicle datasets like nuScenes or Waymo Open, which contain negligible accident data, Accid3nD is specifically curated for hazardous scenarios including high-speed crashes (up to 180 km/h), vehicle fires, and rollovers.

Visualization of the Accid3nD dataset showing 3D bounding boxes, tracking IDs, and trajectories across various lighting and weather conditions.

The data is captured using a multi-modal sensor suite mounted on gantry bridges, including four high-definition cameras, four radars, and a LiDAR system operating at 25 Hz. This infrastructure-based approach offers a "God's eye view," significantly reducing occlusions compared to vehicle-mounted sensors and allowing for continuous tracking of accident development.

Methodology: Building the Digital Twin

To transform raw sensor data into a usable digital twin, the authors developed a multi-stage processing pipeline that integrates perception, tracking, and incident detection.

The technical pipeline for the Accid3nD dataset, from raw sensor input to 3D object detection, multi-sensor fusion, and dual-module accident detection.
  1. 3D Perception & Tracking: Initial vehicle localization is performed using MonoDet3D, followed by PolyMOT for multi-object tracking to maintain consistent identities across frames.
  2. Sensor Fusion: Data from roadside cameras and radars are fused to ensure robustness in challenging weather.
  3. Accident Detection: A dual-module approach is employed:
  4. Rule-based: Uses logic like sudden deceleration (> 5 m/s²) or trajectory deviation (> 30°) for fast, interpretable detection.
  5. Learning-based: Employs a fine-tuned YOLOv8 model to detect specific visual cues like overturned vehicles or fire.

Key Insights and Results

The research highlights several critical findings for infrastructure-based traffic monitoring:

  • Detection Performance: The learning-based YOLOv8x-seg model achieved a [email protected] of 80.10. When combined with MonoDet3D and PolyMOT, the system reached a 3D mAP of 18.24.
  • Speed vs. Accuracy: While the learning-based model is more accurate (F1-score of 0.889), the rule-based system is 57 times faster (running at 95.05 FPS), making it ideal for real-time edge deployment to trigger immediate emergency alerts.
  • Real-World Physics: The dataset captures extreme dynamics—such as vehicles rolling over multiple times or fires—that are difficult to model accurately in simulation.

Conclusion and Future Work

The TUM Traffic Accid3nD dataset provides a vital resource for training safer AD systems by focusing on the critical long-tail events that cause fatalities. Future work aims to expand this network to urban intersections to include vulnerable road users like pedestrians and cyclists, and to increase the diversity of weather and nighttime samples. By bridging the gap between nominal driving data and rare accidents, this work moves the industry closer to the goal of Vision Zero.

Reference: Zimmer, W., et al. "Towards Vision Zero: The TUM Traffic Accid3nD Dataset." accident-dataset.github.io (2025).

Summary updated

Abstract

Even though a significant amount of work has been done to increase the safety of transportation networks, accidents still occur regularly. They must be understood as unavoidable and sporadic outcomes of traffic networks. No public dataset contains 3D annotations of real-world accidents recorded from roadside camera and LiDAR sensors. We present the TUM Traffic Accid3nD (TUMTraf-Accid3nD) dataset, a collection of real-world highway accidents in different weather and lighting conditions. It contains vehicle crashes at high-speed driving with 2,634,233 labeled 2D bounding boxes, instance masks, and 3D bounding boxes with track IDs. In total, the dataset contains 111,945 labeled image and point cloud frames recorded from four roadside cameras and LiDARs at 25 Hz. The dataset contains six object classes and is provided in the OpenLABEL format. We propose an accident detection model that combines a rule-based approach with a learning-based one. Experiments and ablation studies on our dataset show the robustness of our proposed method. The dataset, model, and code are available on our website: https://accident-dataset.github.io.
BibTeX
@inproceedings{zimmer2025towards,
  author = {Walter Zimmer and Ross Greer and Daniel Lehmberg and Marc Pavel and Holger Caesar and Xingcheng Zhou and Ahmed Alaaeldin Ghita and Mohan M. Trivedi and Rui Song and Hu Cao and Akshay Gopalkrishnan and Alois C. Knoll},
  title = {{Towards Vision Zero: The Accid3nD Dataset}},
  booktitle = {Workshops @ ICCV},
  year = {2025},
  url = {https://openreview.net/forum?id=DBE72nS53f},
}
Figure