Abstract
We propose a brand new benchmark for analyzing causality in traffic accident videos by decomposing an accident into a pair of events, cause and effect. We collect videos containing traffic accident scenes and annotate cause and effect events for each accident with their temporal intervals and semantic labels; such annotations are not available in existing datasets for accident anticipation task. Our dataset has the following two advantages over the existing ones, which would facilitate practical research for causality analysis. First, the decomposition of an accident into cause and effect events provides atomic cues for reasoning on a complex environment and planning future actions. Second, the prediction of cause and effect in an accident makes a system more interpretable to humans, which mitigates the ambiguity of legal liabilities among agents engaged in the accident. Using the proposed dataset, we analyze accidents by localizing the temporal intervals of their causes and effects and classifying the semantic labels of the accidents. The dataset as well as the implementations of baseline models are available in the code repository ( https://github.com/tackgeun/CausalityInTrafficAccident ).
Introduction
Most current research in autonomous driving focuses on accident anticipation—predicting that a collision is imminent a few seconds before it happens. While useful, this approach is often a "black box"; it doesn't explain the underlying behavior leading to the crash. This lack of interpretability is a significant hurdle for handling legal liabilities and insurance claims, where distinguishing between the cause (e.g., a car running a red light) and the effect (e.g., a side-impact collision) is crucial.
Furthermore, planning safe maneuvers requires understanding specific "pre-crash" behaviors. Existing datasets often lack the fine-grained temporal and semantic annotations needed to train models on these causal relationships. To address this, we introduce the CTA (Causality in Traffic Accident) benchmark, which treats causality recognition as a temporal action localization task.
The CTA Dataset
The primary contribution of this work is the CTA (Causality in Traffic Accident) dataset, consisting of 1,935 videos (approximately 9.53 hours) sourced from YouTube dashcam and monitoring camera footage. Unlike previous datasets (like Street Accident or NIDB) which only provide generic "accident" labels, CTA provides dual-event intervals and causal labels.
Each video is annotated with a pair of events:
- Cause: 18 categories (e.g., "Running red light," "Control loss").
- Effect: 7 categories (e.g., "Collision with vehicle," "Rollover").
The taxonomy is based on the 2004 General Estimates System (GES) precrash typology, ensuring alignment with real-world police reporting standards. Analysis of the dataset shows that 80.1% of accidents involve collisions between two vehicles, reflecting the most common real-world scenario.
Methodology
The paper treats causality recognition as a temporal action localization task. Given a video, a model must identify the start and end times for both the cause and the effect while simultaneously assigning them semantic labels. Several baseline architectures were evaluated:
- Temporal Segment Networks (TSN): Used for action classification.
- Single-Stream Temporal Action Proposals (SST): A detection-based approach using Gated Recurrent Units (GRUs).
- R-C3D: A 3D convolutional network that generates and classifies activity proposals.
- Multi-Stage Temporal Convolutional Network (MS-TCN): A segmentation-based approach using 1D dilated convolutions to capture long-range dependencies.
A key technical finding was that "linear" consensus in TSN (which preserves temporal order) significantly outperformed "average" consensus, proving that the sequence of frames is vital for causal reasoning. Additionally, models performed better when localizing the effect if they could "look back" from future frames (using backward or bi-directional GRUs), suggesting the aftermath contains vital cues for identifying the crash itself.
Experimental Results
The experimental evaluation yielded several insights:
- Classification: Using a TSN with linear consensus achieved a 59.4% mean Top-1 accuracy across cause and effect labels, significantly beating trivial baselines.
- Localization: The MS-TCN model emerged as the top performer for temporal localization. At a temporal Intersection over Union (tIoU) threshold of 0.5, it achieved 32.97% mean accuracy.
- Segmentation vs. Detection: Segmentation-based models (MS-TCN) generally outperformed detection-based models (R-C3D). This is likely because accidents are continuous processes where frame-level dense prediction captures the transition from risky behavior to physical damage more accurately than discrete bounding boxes.
Conclusion and Future Work
The CTA benchmark provides a structured way to analyze traffic accidents through the lens of causality. While it offers a significant step forward in interpretability, limitations remain. The dataset currently focuses on single-agent causes, whereas real-world pileups may involve multiple contributing factors. Future work should incorporate spatio-temporal tracking to identify specific agents and address the ambiguity in temporal boundaries between cause and effect.
Summary updated