Introduction
The pursuit of "Vision Zero"—a future with zero traffic fatalities—requires autonomous driving (AD) systems to handle not just everyday driving, but the "long-tail" of rare, high-risk events. Most existing AD datasets focus on nominal driving, leaving a critical gap in real-world accident data. Synthetic datasets attempt to fill this gap but suffer from a "sim-to-real" domain gap, while 2D CCTV datasets lack the depth information needed for precise 3D perception.
The TUM Traffic Accid3nD dataset addresses these limitations. It is the first public, large-scale collection of real-world highway accidents captured with 3D annotations from roadside infrastructure, providing a "digital twin" of traffic to study hazardous incidents as they occur.
The Accid3nD Dataset
The dataset comprises 111,945 labeled frames and over 2.6 million 3D bounding boxes. Unlike ego-vehicle datasets like nuScenes or Waymo Open, which contain negligible accident data, Accid3nD is specifically curated for hazardous scenarios including high-speed crashes (up to 180 km/h), vehicle fires, and rollovers.
The data is captured using a multi-modal sensor suite mounted on gantry bridges, including four high-definition cameras, four radars, and a LiDAR system operating at 25 Hz. This infrastructure-based approach offers a "God's eye view," significantly reducing occlusions compared to vehicle-mounted sensors and allowing for continuous tracking of accident development.
Methodology: Building the Digital Twin
To transform raw sensor data into a usable digital twin, the authors developed a multi-stage processing pipeline that integrates perception, tracking, and incident detection.
- 3D Perception & Tracking: Initial vehicle localization is performed using MonoDet3D, followed by PolyMOT for multi-object tracking to maintain consistent identities across frames.
- Sensor Fusion: Data from roadside cameras and radars are fused to ensure robustness in challenging weather.
- Accident Detection: A dual-module approach is employed:
- Rule-based: Uses logic like sudden deceleration (> 5 m/s²) or trajectory deviation (> 30°) for fast, interpretable detection.
- Learning-based: Employs a fine-tuned YOLOv8 model to detect specific visual cues like overturned vehicles or fire.
Key Insights and Results
The research highlights several critical findings for infrastructure-based traffic monitoring:
- Detection Performance: The learning-based YOLOv8x-seg model achieved a [email protected] of 80.10. When combined with MonoDet3D and PolyMOT, the system reached a 3D mAP of 18.24.
- Speed vs. Accuracy: While the learning-based model is more accurate (F1-score of 0.889), the rule-based system is 57 times faster (running at 95.05 FPS), making it ideal for real-time edge deployment to trigger immediate emergency alerts.
- Real-World Physics: The dataset captures extreme dynamics—such as vehicles rolling over multiple times or fires—that are difficult to model accurately in simulation.
Conclusion and Future Work
The TUM Traffic Accid3nD dataset provides a vital resource for training safer AD systems by focusing on the critical long-tail events that cause fatalities. Future work aims to expand this network to urban intersections to include vulnerable road users like pedestrians and cyclists, and to increase the diversity of weather and nighttime samples. By bridging the gap between nominal driving data and rare accidents, this work moves the industry closer to the goal of Vision Zero.
Reference: Zimmer, W., et al. "Towards Vision Zero: The TUM Traffic Accid3nD Dataset." accident-dataset.github.io (2025).
Summary updated