Abstract
Autonomous vehicle (AV) safety relies on sophisticated situational awareness—the ability to interpret complex environmental contexts beyond simple object detection. This paper presents an integrated driving scene analysis system that combines multi-task neural networks (Multi-Net), object detection (YOLO v3), and semantic segmentation (DeepLab v3). By introducing two novel annotated datasets for crash likelihood and road functional classification, the system simultaneously evaluates environmental conditions and identifies "risky" pedestrians within the vehicle's path. Experimental results demonstrate high classification accuracy, though computational overhead remains a challenge for real-time deployment.
Introduction
While AV technology has advanced rapidly, many systems struggle with holistic situational awareness. Isolated tasks, such as detecting a vehicle or a lane line, are insufficient for comprehensive risk assessment. Understanding whether a road is an interstate or a local street, or whether a pedestrian is safely on a sidewalk versus in a drivable lane, is critical for crash prevention.
This research addresses two primary gaps in the field: the lack of integrated methods to extract multiple streams of contextual information from a single video feed, and the scarcity of large-scale datasets focused on crash indicators and road functional classifications (e.g., arterial vs. collector roads).
Methodology
The proposed modular architecture processes RGB dashcam images through three primary components to generate a comprehensive scene analysis.
Multi-Net Architecture
The core novelty lies in the Multi-Net, which consists of two parallel multi-task networks:
- Crash and Road Classification: Uses a multi-level perceptron (MLP) to classify crash likelihood (no crash, pre-crash, crash) and road function (arterial, collector, interstate, local).
- Environmental Classification: A Convolutional Neural Network (CNN) that identifies weather conditions and time of day.
Object Detection and Segmentation
YOLO v3 is employed for real-time detection of vehicles and pedestrians. To move beyond simple detection, the system integrates DeepLab v3 for semantic segmentation to define the "drivable area."
A key feature of the system is the "Risky Pedestrian" logic. Instead of flagging all pedestrians, the system checks for spatial overlaps between YOLO-detected pedestrian bounding boxes and the DeepLab-segmented drivable lane. Pedestrians located within the lane are prioritized as high-risk targets.
Dataset Development
The study utilizes a combination of existing and custom-built data to train these specialized models:
- Crash Likelihood Dataset: 15,900 images mined from YouTube and the HDD dataset, manually labeled for crash stages (with "pre-crash" defined as the two-second window before impact).
- Road Function Dataset: 6,400 images labeled according to U.S. Federal Highway Administration (FHWA) functional systems.
- BDD100k & Synthetic Data: Used for weather and time-of-day classification. To address data scarcity in foggy conditions, the authors generated synthetic foggy images to balance the training set.
Experimental Results
The system demonstrated robust performance across its primary classification tasks. The Multi-Net achieved a macro-average F1 score of 93% for crash likelihood and 92% for road function. Performance for weather (74%) and time of day (85%) was lower, primarily due to statistical similarities in pixel distributions between categories like "clear" and "snowy."
In terms of computational efficiency, the Multi-Net and YOLO v3 components achieved approximately 6 fps. However, the inclusion of the DeepLab v3 segmentation model created a significant bottleneck, reducing the overall system speed to 2 fps.
Conclusion and Future Work
This research provides a framework for multi-task driving scene analysis that significantly enhances situational awareness. While the classification accuracy is high, the current 2 fps inference speed is insufficient for high-speed autonomous driving. Future work will focus on optimizing the semantic segmentation component with lighter models and exploring shared architectures to reduce memory overhead and improve real-time performance.
Summary updated