DRIVE: Deep Reinforced Accident Anticipation with Visual Explanation

Wentao Bao, Qi Yu, Yu Kong

Introduction: The Challenge of Accident Anticipation

Predicting a traffic accident before it occurs is a critical goal for autonomous driving safety. However, existing AI models face two primary hurdles: the Early-vs-Correct Trade-off and the "Black Box" Problem. Early predictions often lack sufficient visual cues, leading to unreliability, while most models fail to provide human-interpretable reasons for their warnings.

The DRIVE framework addresses these issues by treating accident anticipation as a Markov Decision Process (MDP). In this setup, an agent learns to explore the visual environment to maximize a reward based on both timing and accuracy, effectively learning to "look ahead" for potential hazards.

The DRIVE Framework: Architecture and Methodology

The core of the DRIVE model (Deep ReInforced accident anticipation with Visual Explanation) is a multi-task reinforcement learning architecture designed to simulate human-like visual processing.

The DRIVE architecture featuring the Traffic Observation Environment and Stochastic Multi-task Agent.

The architecture consists of several key components:

  • Stochastic Multi-task Agent: This agent simultaneously predicts accident probability and determines the "fixation point" (the focus of attention) for the next frame.
  • Dynamic Attention Fusion (DAF): DRIVE combines bottom-up attention (noticing salient movement or lights) with top-down attention (searching for task-specific risks). As danger increases, the model shifts focus toward top-down risky regions.
  • Soft Actor-Critic (SAC) Learning: The model uses an adapted SAC algorithm with a Dense Anticipation Reward to encourage early detection and a Sparse Fixation Reward to align the agent's focus with human driver eye-tracking data.
  • Regularized Auto-Encoder (RAE): An RAE compresses high-dimensional video data into a latent representation, making RL training more efficient.

Datasets and Evaluation

The model was evaluated on two major benchmarks:

  1. DADA-2000: Contains 2,000 accident videos with precise timing and human eye-fixation annotations.
  2. DAD (Dashcam Accident Dataset): A widely used dataset that lacks fixation data, used here to demonstrate DRIVE's performance even without explicit "where to look" labels during training.

Results and Visual Explanations

DRIVE significantly outperforms previous state-of-the-art (SOTA) methods in both accuracy and earliness. On the DADA-2000 dataset, it achieved an AUC of 72.27% (up from ~60%), and on the DAD dataset, it reached 93.82% AUC. Crucially, it anticipates crashes roughly 3.6 to 3.7 seconds before impact.

Beyond metrics, DRIVE provides introspective explanations through its attention maps. By forcing the model to predict a fixation point, the resulting saliency maps show exactly which objects triggered the alarm.

Comparison of full frame, foveal frame, and the two types of attention (bottom-up and top-down) used by DRIVE.

The use of foveal vision allows the model to filter out background noise and focus on the causal elements of an accident, such as a pedestrian entering a crosswalk or a vehicle merging blindly. This demonstrates that visual attention is not just a byproduct but a core component of the prediction logic.

Limitations and Future Directions

While DRIVE represents a major advancement, challenges remain. Current reward functions treat false positives and false negatives similarly, whereas real-world safety requires prioritizing the avoidance of false negatives (missed accidents). Additionally, the computational overhead of dual-pathway attention and foveal vision must be optimized for real-time deployment on vehicle edge hardware. Future work will also explore learning effective attention patterns in the absence of high-quality human eye-tracking data.

Summary updated

Abstract

Traffic accident anticipation aims to accurately and promptly predict the occurrence of a future accident from dashcam videos, which is vital for a safety-guaranteed self-driving system. To encourage an early and accurate decision, existing approaches typically focus on capturing the cues of spatial and temporal context before a future accident occurs. However, their decision-making lacks visual explanation and ignores the dynamic interaction with the environment. In this paper, we propose Deep ReInforced accident anticipation with Visual Explanation, named DRIVE. The method simulates both the bottom-up and top-down visual attention mechanism in a dashcam observation environment so that the decision from the proposed stochastic multi-task agent can be visually explained by attentive regions. Moreover, the proposed dense anticipation reward and sparse fixation reward are effective in training the DRIVE model with our improved reinforcement learning algorithm. Experimental results show that the DRIVE model achieves state-of-the-art performance on multiple real-world traffic accident datasets. Code and pre-trained model are available at \url{https://www.rit.edu/actionlab/drive}.
BibTeX
@inproceedings{bao2021drive,
  author = {Wentao Bao and Qi Yu and Yu Kong},
  title = {{DRIVE: Deep Reinforced Accident Anticipation with Visual Explanation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00752},
}
Figure