TL;DR
This paper introduces DEEPCRASHTEST, a novel framework that automatically extracts 3D vehicle trajectories from dashcam videos of real-world crashes. These extracted trajectories are then used to recreate these hazardous scenarios in a simulator, enabling more robust training and testing of autonomous driving systems without the need for dangerous real-world experimentation.
What Problem Does This Paper Address?
Training and validating autonomous vehicles (AVs) requires vast amounts of data, especially for rare and dangerous scenarios like accidents. While companies collect data from real-world driving, replicating crash situations is inherently unsafe and ethically problematic. Existing methods for generating simulated crash scenarios often rely on manually engineered environments or incomplete data from police reports. This paper tackles the challenge of automatically extracting detailed, real-world crash scenarios from readily available dashcam videos to create realistic virtual crash tests.
Methods
DEEPCRASHTEST employs a modular pipeline that leverages existing deep learning components to extract 3D vehicle trajectories from monocular dashcam footage. The core steps include:
- Vehicle Detection and Tracking: Using Mask R-CNN for instance segmentation and object detection, followed by the Re3-Tracker for robust tracking, and the Hungarian algorithm for data association.
- 3D Bounding Box Estimation: Employing the 3D-Deepbox network to estimate the 3D dimensions and orientation of vehicles.
- Point Cloud Method for 3D Vehicle Centers: An alternative approach using monocular depth estimation (Monodepth) to generate a point cloud and approximate vehicle positions.
- Ego Motion Estimation: Utilizing monocular visual odometry and optical flow-based approaches to estimate the ego vehicle's motion, crucial for determining absolute trajectories.
- Lane Detection and Tracking: Employing the deep learning-based LaneNet for robust lane detection, followed by DBSCAN and B-spline fitting for lane ID assignment and tracking.
- Camera Calibration: Estimating camera parameters, either by using pre-trained dataset parameters or by reformulating perspective camera equations using lane line annotations.
- Trajectory Smoothing and Simulation: Raw trajectories are smoothed using Savitzky-Golay filters and splines. These smoothed trajectories are then fed into the Sim-ATAV simulator to recreate the crash scenarios. The framework also includes methods for extrapolating trajectories and generating road waypoints to ensure plausible simulations.
Datasets
The paper utilizes the KITTI dataset for quantitative evaluation of the trajectory extraction module. They note that the KITTI benchmark typically provides tracking evaluation relative to the ego vehicle, but they modify this by adding ground truth odometry to evaluate absolute trajectory positions.
Key Insights and Ideas
- Leveraging Internet-Scale Data: The proliferation of dashcams provides an abundant, albeit noisy, source of real-world crash data that can be ethically and safely exploited for AV training.
- Modular Approach for Robustness: Breaking down the complex problem into smaller, manageable sub-problems, each addressed by a specialized deep learning component, allows for better generalization and easier integration of future improvements.
- Bridging the Gap Between Video and Simulation: The framework effectively translates visual information from dashcam videos into actionable data for vehicle simulators, enabling the creation of realistic and challenging test cases.
- Importance of Ego Motion: Accurately estimating the ego vehicle's motion is critical for transforming relative object trajectories into absolute world-frame trajectories, which are essential for simulation.
- Qualitative Evaluation is Key: For this application, visual inspection and qualitative correctness of the simulated crash scenarios are deemed sufficient, acknowledging that perfect quantitative accuracy is not always necessary for effective testing.
Key Results
- The paper demonstrates the ability to extract 3D vehicle trajectories from dashcam videos and recreate them in the Webots simulator using the Sim-ATAV framework.
- Qualitative evaluations show the successful recreation of crash scenarios, including a head-on collision on a sharp turn.
- The framework was tested on three KITTI tracking sequences (3, 8, and 10), achieving respectable tracking evaluation results, though the authors note potential slight performance degradation due to camera calibration errors.
- The system successfully generated "safe" maneuvers by exploring variations in initial conditions, demonstrating the potential for testing collision avoidance systems.
Limitations and Future Work
- Video Length Constraint: Videos are generally limited to under one minute to mitigate drift in position estimates caused by monocular sensor effects.
- Scene Simplicity: The approach is primarily focused on straight or curved highway scenarios, with less emphasis on complex intersections due to limited agent visibility.
- Camera Assumption: The camera is assumed to be attached to the ego car's dashboard, though manual adjustments can relax this.
- Accuracy of Depth Estimation: The paper acknowledges that the 3D coordinates derived from depth maps are approximations, as they only represent visible parts of the vehicle.
- Future Work: The authors suggest that DEEPCRASHTEST can be extended to other simulation-based automated test generation tools for AVs. They also imply that further refinement of individual modules and potentially end-to-end approaches could improve performance.
Summary updated