A system of vision sensor based deep neural networks for complex driving scene analysis in support of crash risk assessment and prevention

Muhammad Monjurul Karim, Yu Li, Ruwen Qin, Zhaozheng YinarXiv 2021 · arXiv

Abstract

Autonomous vehicle (AV) safety relies on sophisticated situational awareness—the ability to interpret complex environmental contexts beyond simple object detection. This paper presents an integrated driving scene analysis system that combines multi-task neural networks (Multi-Net), object detection (YOLO v3), and semantic segmentation (DeepLab v3). By introducing two novel annotated datasets for crash likelihood and road functional classification, the system simultaneously evaluates environmental conditions and identifies "risky" pedestrians within the vehicle's path. Experimental results demonstrate high classification accuracy, though computational overhead remains a challenge for real-time deployment.

Introduction

While AV technology has advanced rapidly, many systems struggle with holistic situational awareness. Isolated tasks, such as detecting a vehicle or a lane line, are insufficient for comprehensive risk assessment. Understanding whether a road is an interstate or a local street, or whether a pedestrian is safely on a sidewalk versus in a drivable lane, is critical for crash prevention.

This research addresses two primary gaps in the field: the lack of integrated methods to extract multiple streams of contextual information from a single video feed, and the scarcity of large-scale datasets focused on crash indicators and road functional classifications (e.g., arterial vs. collector roads).

Methodology

The proposed modular architecture processes RGB dashcam images through three primary components to generate a comprehensive scene analysis.

Figure 1: The integrated system architecture utilizing DeepLab v3, YOLO v3, and Multi-Net for holistic driving scene analysis.

Multi-Net Architecture

The core novelty lies in the Multi-Net, which consists of two parallel multi-task networks:

  1. Crash and Road Classification: Uses a multi-level perceptron (MLP) to classify crash likelihood (no crash, pre-crash, crash) and road function (arterial, collector, interstate, local).
  2. Environmental Classification: A Convolutional Neural Network (CNN) that identifies weather conditions and time of day.

Object Detection and Segmentation

YOLO v3 is employed for real-time detection of vehicles and pedestrians. To move beyond simple detection, the system integrates DeepLab v3 for semantic segmentation to define the "drivable area."

A key feature of the system is the "Risky Pedestrian" logic. Instead of flagging all pedestrians, the system checks for spatial overlaps between YOLO-detected pedestrian bounding boxes and the DeepLab-segmented drivable lane. Pedestrians located within the lane are prioritized as high-risk targets.

Dataset Development

The study utilizes a combination of existing and custom-built data to train these specialized models:

  • Crash Likelihood Dataset: 15,900 images mined from YouTube and the HDD dataset, manually labeled for crash stages (with "pre-crash" defined as the two-second window before impact).
  • Road Function Dataset: 6,400 images labeled according to U.S. Federal Highway Administration (FHWA) functional systems.
  • BDD100k & Synthetic Data: Used for weather and time-of-day classification. To address data scarcity in foggy conditions, the authors generated synthetic foggy images to balance the training set.

Experimental Results

The system demonstrated robust performance across its primary classification tasks. The Multi-Net achieved a macro-average F1 score of 93% for crash likelihood and 92% for road function. Performance for weather (74%) and time of day (85%) was lower, primarily due to statistical similarities in pixel distributions between categories like "clear" and "snowy."

Figure 5: Visualization of the "Risky Pedestrian" logic, where pedestrians in the drivable lane are highlighted and labeled.

In terms of computational efficiency, the Multi-Net and YOLO v3 components achieved approximately 6 fps. However, the inclusion of the DeepLab v3 segmentation model created a significant bottleneck, reducing the overall system speed to 2 fps.

Conclusion and Future Work

This research provides a framework for multi-task driving scene analysis that significantly enhances situational awareness. While the classification accuracy is high, the current 2 fps inference speed is insufficient for high-speed autonomous driving. Future work will focus on optimizing the semantic segmentation component with lighter models and exploring shared architectures to reduce memory overhead and improve real-time performance.

Summary updated

Abstract

To assist human drivers and autonomous vehicles in assessing crash risks, driving scene analysis using dash cameras on vehicles and deep learning algorithms is of paramount importance. Although these technologies are increasingly available, driving scene analysis for this purpose still remains a challenge. This is mainly due to the lack of annotated large image datasets for analyzing crash risk indicators and crash likelihood, and the lack of an effective method to extract lots of required information from complex driving scenes. To fill the gap, this paper develops a scene analysis system. The Multi-Net of the system includes two multi-task neural networks that perform scene classification to provide four labels for each scene. The DeepLab v3 and YOLO v3 are combined by the system to detect and locate risky pedestrians and the nearest vehicles. All identified information can provide the situational awareness to autonomous vehicles or human drivers for identifying crash risks from the surrounding traffic. To address the scarcity of annotated image datasets for studying traffic crashes, two completely new datasets have been developed by this paper and made available to the public, which were proved to be effective in training the proposed deep neural networks. The paper further evaluates the performance of the Multi-Net and the efficiency of the developed system. Comprehensive scene analysis is further illustrated with representative examples. Results demonstrate the effectiveness of the developed system and datasets for driving scene analysis, and their supportiveness for crash risk assessment and crash prevention.
BibTeX
@misc{karim2021system,
  author = {Muhammad Monjurul Karim and Yu Li and Ruwen Qin and Zhaozheng Yin},
  title = {{A system of vision sensor based deep neural networks for complex driving scene analysis in support of crash risk assessment and prevention}},
  year = {2021},
  eprint = {2106.10319},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.10319},
}
Figure