Cognitive Accident Prediction in Driving Scenes: A Multimodality Benchmark

Jianwu Fang, Lei-Lei Li, Kuan Yang, Zhedong Zheng, Jianru Xue, Tat-Seng ChuaarXiv 2022 · DOI · arXiv

Introduction

Predicting traffic accidents from dashcam footage remains a significant challenge in autonomous driving and advanced driver-assistance systems (ADAS). These events are inherently "long-tail"—rare, highly varied, and often occurring under adverse conditions such as heavy rain, night-time glare, or complex urban environments. Traditional state-of-the-art models typically rely on object tracking (e.g., monitoring cars and pedestrians). However, these systems often fail when visibility is low or when objects are partially obscured, leading to a breakdown in prediction.

Humans do not merely track pixels; we employ cognition. We interpret the context of a scene, recognize dangerous behaviors (like a vehicle weaving), and focus our attention on high-risk zones. This research introduces the Cognitive Accident Prediction (CAP) framework, designed to bridge the gap between raw visual processing and human-like reasoning to achieve a higher Time-to-Accident (TTA)—the critical window needed for early warning systems to prevent collisions.

The CAP Framework

The CAP model integrates three specialized modules to mimic human driving intuition:

  1. Attentive Text-to-Vision Shift Fusion: During the training phase, the model is provided with factual text descriptions of the scene. Using a novel "shift" mechanism (T2I-SFLayer), the model learns to associate visual patterns with semantic concepts. At test time, when no text is available, the model "shifts" this learned linguistic knowledge into its visual path to enhance inference.
  2. Attentive Semantic Context Transfer: The scene is modeled as a graph (GCN) to capture spatial relationships between entities, which is then processed by a Gated Recurrent Unit (GRU) to learn the temporal progression of the accident.
  3. Driver Attention-Guided Prediction: The model is trained to reconstruct human driver fixation maps. This acts as a spatial prior, forcing the AI to prioritize the same regions a human would monitor.
Driver attention maps showing the focus on a pedestrian before and during a collision.

CAP-DATA: A New Standard for Accident Benchmarking

To support this cognitive approach, the researchers introduced CAP-DATA, the largest and most richly annotated dataset of its kind. While previous datasets often provided simple binary labels (accident/no accident), CAP-DATA offers a comprehensive fact-effect-reason-introspection framework.

  • Scale: 11,727 in-the-wild accident videos comprising 2.19 million frames.
  • Rich Annotation: Each video includes detailed descriptions, such as the Fact (a vehicle changes lanes), the Effect (ego-car hits a truck), the Reason (front truck didn't use a turn signal), and Introspection (ego-car should have driven more slowly).
The CAP-DATA annotation framework, including factual, causal, and preventative descriptions across a temporal window.

Key Insights and Results

The study yielded several critical findings regarding the role of multimodality in safety systems:

  • Text as a Semantic Teacher: Text provides a dense semantic guide that visual pixels alone lack. By training with linguistic data, the model learns what to look for, such as a "pedestrian crossing," even when the visual signal is noisy or degraded.
  • Attention as Traction: Reconstructing driver attention helps the model ignore irrelevant background noise (like billboards) and focus on the "crashing object," even if it is small or poorly lit.
  • Performance Gains: The CAP model significantly outperformed previous benchmarks like DRIVE and DSA-RNN. On the MINI-test set, CAP achieved a TTA of 4.02 seconds, a substantial improvement over the 3.65 seconds achieved by the previous best model. In high-speed scenarios, this extra 0.37 seconds can be the difference between a successful emergency brake and a collision. Furthermore, CAP reached an AUC (Area Under Curve) of 0.83, compared to 0.69 for DRIVE.

Limitations and Future Directions

Despite its success, the CAP model relies on high-quality training data, which is labor-intensive to annotate. Additionally, while the model can predict that an accident will occur, it does not yet perform counterfactual reasoning—the ability to simulate alternative outcomes (e.g., "What if I steered left?"). Future research will focus on cause-effect problems and the integration of diverse perspectives, such as drone-based navigation maps, to provide a "bird's eye" cognitive layer to the dashcam's perspective.

Summary updated

Abstract

Traffic accident prediction in driving videos aims to provide an early warning of the accident occurrence, and supports the decision making of safe driving systems. Previous works usually concentrate on the spatial-temporal correlation of object-level context, while they do not fit the inherent long-tailed data distribution well and are vulnerable to severe environmental change. In this work, we propose a Cognitive Accident Prediction (CAP) method that explicitly leverages human-inspired cognition of text description on the visual observation and the driver attention to facilitate model training. In particular, the text description provides a dense semantic description guidance for the primary context of the traffic scene, while the driver attention provides a traction to focus on the critical region closely correlating with safe driving. CAP is formulated by an attentive text-to-vision shift fusion module, an attentive scene context transfer module, and the driver attention guided accident prediction module. We leverage the attention mechanism in these modules to explore the core semantic cues for accident prediction. In order to train CAP, we extend an existing self-collected DADA-2000 dataset (with annotated driver attention for each frame) with further factual text descriptions for the visual observations before the accidents. Besides, we construct a new large-scale benchmark consisting of 11,727 in-the-wild accident videos with over 2.19 million frames (named as CAP-DATA) together with labeled fact-effect-reason-introspection description and temporal accident frame label. Based on extensive experiments, the superiority of CAP is validated compared with state-of-the-art approaches. The code, CAP-DATA, and all results will be released in \url{https://github.com/JWFanggit/LOTVS-CAP}.
BibTeX
@misc{fang2022cognitive,
  author = {Jianwu Fang and Lei-Lei Li and Kuan Yang and Zhedong Zheng and Jianru Xue and Tat-Seng Chua},
  title = {{Cognitive Accident Prediction in Driving Scenes: A Multimodality Benchmark}},
  year = {2022},
  eprint = {2212.09381},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.09381},
}
Figure