End-to-end pedestrian collision warning system based on a convolutional neural network with semantic segmentation

Heechul Jung, Min-Kook Choi, Soon Kwon, Woo Young Jung

Introduction

Pedestrian Collision Warning (PCW) systems are critical components of modern Advanced Driver Assistance Systems (ADAS). However, existing solutions often struggle with high false alarm rates, leading to driver annoyance and a lack of trust. This paper presents an end-to-end approach that leverages a single Convolutional Neural Network (CNN) to perform both pedestrian detection and semantic scene understanding, significantly improving the reliability of collision warnings.

The Limitations of Traditional Pipelines

Traditional PCW systems typically employ a multi-stage pipeline. As illustrated below, these systems first detect pedestrians using hand-crafted features, such as Histograms of Oriented Gradients (HoG), and then apply a decision rule based on whether the detection falls within a predefined "dangerous" region of interest.

Traditional PCW systems rely on a sequential pipeline of detection and region-based decision making, often leading to high false alarm rates.

This approach suffers from two primary weaknesses:

  1. High False Alarm Rates: These systems often trigger warnings for pedestrians who are safely on the sidewalk, as they lack the semantic context to distinguish between a pedestrian in the road and one on a path.
  2. Pipeline Fragility: Because the detection and decision stages are separate, errors in the initial detection stage—caused by lighting, occlusion, or unusual poses—propagate through the system, resulting in total failure.

Proposed Architecture: Multi-Task Learning

To address these issues, the authors propose a unified CNN architecture that maps raw pixels directly to a collision warning. The core innovation lies in the integration of a semantic segmentation network with the primary prediction network.

The proposed multi-task CNN architecture integrates a prediction network with a semantic segmentation branch, sharing early convolutional layers to learn context-aware features.

The network shares its initial layers (CONV1 and CONV2) across both tasks. This allows the model to learn fundamental visual features that are useful for both identifying objects and understanding the scene layout. High-level features from the segmentation branch (FC3) are then concatenated with features from the prediction branch (FC1) before the final decision layer (FC2). This integration ensures that the "danger" prediction is informed by the semantic context—for example, recognizing that a pedestrian is located on a "sidewalk" rather than the "road."

Experimental Setup and Datasets

The researchers utilized the Cityscapes dataset, which provides high-resolution urban imagery and dense semantic annotations. Because Cityscapes lacks specific collision warning labels, the authors manually annotated approximately 4,500 images with binary "warning" or "no warning" labels. To handle the inherent class imbalance (where safe scenarios are far more frequent than dangerous ones), the training process employed oversampling of the "warning" class.

Key Findings

The proposed end-to-end system was evaluated against a traditional HoG-based baseline and a standalone CNN without the segmentation branch.

  • Superior Accuracy: At a fixed 15% false positive rate, the multi-task CNN achieved 71% accuracy, a significant improvement over the 45% accuracy of the traditional HoG-based system.
  • The Value of Context: Incorporating semantic segmentation improved the CNN's performance from 63.94% to 71%. This demonstrates that the auxiliary task of scene understanding directly enhances the primary task of collision prediction.
  • Robustness: Qualitatively, the system demonstrated an ability to ignore pedestrians on sidewalks and correctly identify other hazards, such as cyclists, without explicit programming for those specific categories.

Conclusion

The transition from a modular, hand-crafted pipeline to an end-to-end, context-aware CNN represents a significant step forward for PCW systems. By teaching the network "what" and "where" objects are through semantic segmentation, the system gains the necessary context to distinguish between safe and dangerous pedestrian encounters. This approach not only reduces false alarms but also creates a more robust framework for future ADAS developments.

Summary updated

Abstract

Traditional pedestrian collision warning systems sometimes raise alarms even when there is no danger (e.g., when all pedestrians are walking on the sidewalk). These false alarms can make it difficult for drivers to concentrate on their driving. In this paper, we propose a novel framework for an end-to-end pedestrian collision warning system based on a convolutional neural network. Semantic segmentation information is used to train the convolutional neural network and two loss functions, such as cross entropy and Euclidean losses, are minimized. Finally, we demonstrate the effectiveness of our method in reducing false alarms and increasing warning accuracy compared to a traditional histogram of oriented gradients (HoG)-based system.
BibTeX
@inproceedings{jung2016end,
  author = {Heechul Jung and Min-Kook Choi and Soon Kwon and Woo Young Jung},
  title = {{End-to-end pedestrian collision warning system based on a convolutional neural network with semantic segmentation}},
  booktitle = {ICCE},
  year = {2016},
  doi = {10.1109/icce.2018.8326129},
}
Figure