Agent-Centric Risk Assessment: Accident Anticipation and Risky Region Localization

Kuo-Hao Zeng, Shih-Han Chou, Fu-Hsiang Chan, Juan Carlos Niebles, Min Sun

TL;DR

This paper introduces an agent-centric framework for risk assessment that simultaneously predicts when an accident will happen (anticipation) and where it will occur (localization). By combining a novel dynamic parameter prediction layer with an "imagination" module that forecasts future agent trajectories, the model significantly outperforms existing baselines on both traffic dashcam data and a new "Epic Fail" viral video dataset.

What Problem Does This Paper Address?

Risk is not distributed uniformly across an environment, nor is it static. For a living agent, assessing risk involves two critical sub-tasks: temporal anticipation (knowing a threat is coming) and spatial localization (knowing where the threat is).

Existing computer vision approaches often treat accident prediction as a simple action classification problem or focus solely on environmental priors (e.g., "this intersection is dangerous"). However, risk is highly contextual. A flight of stairs is a mundane architectural feature for a pedestrian but a high-risk region for someone on a unicycle. The authors argue that risk assessment must be agent-centric, focusing on the specific interactions between an agent's behavior and its surroundings.

Methods

The authors propose a novel Recurrent Neural Network (RNN) architecture consisting of three main components:

  1. Agent-Region Interaction Module: Instead of using static weights, this module uses a dynamic parameter predictor. It takes the agent's appearance and its spatial relationship to a candidate region (encoded as a 9D vector including relative position and IoU) to dynamically generate the weights used to score that region's riskiness.
  2. Recurrent Temporal Memory: The system employs two LSTMs. The first (Agent-RNN) tracks the agent's behavior over time. The second (Accident-Anticipation-RNN) aggregates holistic scene information—combining agent data with a weighted sum of all risky regions—to produce a final accident probability.
  3. Imagination Module: Perhaps the most novel feature is the ability to "imagine" the future. The model uses its current hidden state to predict the agent's location $K$ frames into the future. This imagined location is then fed back into the risk assessment pipeline, allowing the model to anticipate accidents even before the agent physically reaches a dangerous spot.
The model architecture showing the interaction between agent info, region proposals, and the imagination module.

Datasets

The researchers evaluated their model on two primary datasets:

  • Street Accident (SA): A collection of dashcam videos from six cities in Taiwan, featuring diverse traffic accidents (car-vs-bike, car-vs-car, etc.).
  • Epic Fail (EF): A new, large-scale dataset introduced in this paper. It consists of 3,000 viral videos of human accidents (e.g., parkour mishaps, skateboarding falls). This dataset is particularly challenging due to the wide variety of agents, environments, and failure types.

Key Insights and Ideas

  • Contextual Risk: The "aha moment" here is that risk is a function of interaction. By using dynamic parameters (Eq. 2 in the paper), the model learns that the same object (like a bench) has different risk profiles depending on whether the agent is walking past it or jumping over it on a bike.
  • Spatial Reasoning: Unlike previous models that might just flag a whole video as "risky," this model produces a spatial Risk Map. This allows it to pinpoint the specific object or area (the "risky region") likely to cause the accident.
  • Forecasting as a Prior: The imagination module acts as a temporal look-ahead. By simulating future trajectories, the model can "see" an accident coming before the visual evidence is fully present in the current frame.
Illustration of the 9D spatial relationship vector used to calculate risk between the agent and surrounding regions.

Key Results

The proposed model, specifically the version with memory and imagination (L-RAI), consistently outperformed baselines like Dynamic Soft-Attention (DSA) and R*CNN.

  • Accident Anticipation: On the EF dataset, L-RAI achieved a mean Average Precision (mAP) of 75.1%, a massive jump over the DSA baseline's 45.7%.
  • Early Warning: The model achieved an Average Time-to-Accident (ATTA) of roughly 2.23 to 3.01 seconds across both datasets, providing a critical window for potential intervention.
  • Localization: While risky region localization is a much harder task, the model achieved 15.1% mAP on the EF dataset, significantly outperforming the R*CNN baseline (3.5%).
Figure 5. Qualitative Results. We set 0.9 as the threshold of triggering accident anticipation to show the qualitative results. In each example, we show a typical example with accident anticipation probability (bottom row), the heat map (yellow for high risk and blue for low risk) for the risky regions (middle row) and ground truth risky region (orange box in top row). For risky heat map, we average risky confidences of covering boxes for each pixel and draw the map by using Matlab [19] imagesc tool. More detail for drawing heat map can be found in Appendix E . The first and second one are positive examples. The third and fourth one are negative examples. The last one is a failure case, where the model misunderstands risky regions so that it has higher accident anticipation probability at first. However, after long-term observation, the model correct the anticipation probability.

Limitations and Future Work

The primary limitation acknowledged by the authors is that the imagination module currently only simulates the agent. It assumes the rest of the environment remains static. In dynamic scenarios like heavy traffic, other vehicles and pedestrians are also moving, which changes the risk landscape.

Future work aims to extend the imagination layer to simulate both the agent and the environment simultaneously, which would provide a more robust and explainable model for complex real-world safety applications.

Summary updated

Abstract

For survival, a living agent must have the ability to assess risk (1) by temporally anticipating accidents before they occur, and (2) by spatially localizing risky regions in the environment to move away from threats. In this paper, we take an agent-centric approach to study the accident anticipation and risky region localization tasks. We propose a novel soft-attention Recurrent Neural Network (RNN) which explicitly models both spatial and appearance-wise non-linear interaction between the agent triggering the event and another agent or static-region involved. In order to test our proposed method, we introduce the Epic Fail (EF) dataset consisting of 3000 viral videos capturing various accidents. In the experiments, we evaluate the risk assessment accuracy both in the temporal domain (accident anticipation) and spatial domain (risky region localization) on our EF dataset and the Street Accident (SA) dataset. Our method consistently outperforms other baselines on both datasets.
BibTeX
@inproceedings{zeng2017agent,
  author = {Kuo-Hao Zeng and Shih-Han Chou and Fu-Hsiang Chan and Juan Carlos Niebles and Min Sun},
  title = {{Agent-Centric Risk Assessment: Accident Anticipation and Risky Region Localization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2017},
  url = {https://openreview.net/forum?id=BJ-euyMOZS},
}
Figure