A Safety-Adapted Loss for Pedestrian Detection in Automated Driving

Maria Lyssenko, Piyush Pimplikar, Maarten Bieshaar, Farzad Nozarian, Rudolph TriebelarXiv 2024 · OpenReview

TL;DR

This paper introduces a novel "safety-adapted focal loss" for pedestrian detection in autonomous driving. By incorporating a "criticality score" for each pedestrian, derived from their predicted collision risk and proximity to the vehicle, the model learns to prioritize detecting pedestrians who pose the greatest danger, thereby reducing critical false negatives without compromising overall detection performance.

What Problem Does This Paper Address?

In automated driving (AD), accurately detecting vulnerable road users (VRUs) like pedestrians is paramount for safety. Traditional object detection metrics, such as mean average precision (mAP), don't adequately capture the safety implications of misdetections. A false negative for a pedestrian in the direct path of an autonomous vehicle (AV) is far more dangerous than one for a pedestrian far away. Current training methods penalize all misdetections equally, failing to prioritize the detection of "safety-critical" pedestrians. This paper aims to bridge this gap by developing a training strategy that explicitly accounts for the varying levels of risk posed by individual pedestrians.

Methods

The core contribution is a novel safety-adapted focal loss function. This loss function modifies the standard focal loss by incorporating a per-pedestrian criticality score (Kᵢ). This score is calculated by combining two factors:

  1. Collision Criticality (K<0xE1><0xB5><0x9C>,ᵢ): Derived from the Reachability Set-Based Time-to-Collision (TTCRSB). This metric uses reachability analysis from the motion domain to estimate the earliest possible time a collision could occur between the AV and a pedestrian, considering their uncertain future movements.
  2. Distance Criticality (K<0xE1><0xB5><0x9A>,ᵢ): Incorporates the physical distance between the pedestrian and the AV. This is crucial because even if the TTCRSB is not immediately critical, a pedestrian close to the AV still poses a significant risk.

The safety-adapted focal loss then uses this criticality score (Kᵢ) to dynamically adjust the loss contribution for each pedestrian. Critically, pedestrians with higher criticality scores have their misdetections penalized more heavily during training, effectively guiding the neural network to focus on them. The authors also propose a specific formulation for Kᵢ, weighting collision criticality more heavily and using a downward parabola for distance criticality to ensure that proximity is strongly considered.

Datasets

  • nuImages: Used for initial pre-training of the pedestrian detectors. This dataset provides 2D bounding box annotations.
  • nuScenes: The primary dataset for training and evaluation. It's a multimodal dataset providing 3D sensor data (including lidar and camera) and detailed annotations for 1000 scenes. The authors project 3D bounding boxes from nuScenes onto the camera view to obtain 2D pedestrian annotations.

Key Insights and Ideas

  • Beyond Generic Metrics: Standard evaluation metrics are insufficient for safety-critical applications like AD. We need metrics that reflect the actual risk posed by misdetections.
  • Motion Domain for Safety: Integrating information from the motion domain (via reachability analysis and TTCRSB) is crucial for accurately assessing collision risk, especially when considering uncertainties in pedestrian and AV movement.
  • Proximity Matters: Even with a seemingly safe TTCRSB, a pedestrian's close proximity to the AV significantly increases their risk, necessitating a combined criticality score.
  • Instance-Level Safety: Safety is not a category-wide property but an instance-specific one. Different pedestrians in the same scene can have vastly different risk profiles.
  • Targeted Training: By adapting the loss function based on individual pedestrian criticality, we can effectively "teach" the model to prioritize the detection of the most dangerous scenarios.

Key Results

The paper demonstrates that training with the safety-adapted focal loss (FL<0xE2><0x82><0x96>) leads to significant improvements in recall for safety-critical pedestrians (Zone C).

  • For RetinaNet, recall in Zone C improved by 2.5% compared to the baseline focal loss with γ=2.
  • For FCOS, recall in Zone C improved by 3.2% compared to the baseline focal loss with γ=2.
  • The safety-adapted loss also showed improvements for potentially critical pedestrians (Zone PC) and maintained stable overall performance, with a negligible degradation of less than 1.0% in overall performance metrics.
  • An ablation study showed that while distance alone can be a useful proxy for criticality, combining it with collision risk (TTCRSB) provides a more robust measure.

Limitations and Future Work

  • False Positives: The current work focuses on mitigating false negatives (missed detections). The authors acknowledge that false positives are also a significant safety concern and suggest future work to determine the criticality of falsely detected pedestrians.
  • Regression Task: The current safety adaptation is applied to the detection loss. Future work aims to extend this to the regression task to improve the localization accuracy of critical pedestrians.
  • Visibility Proxy: While visibility was used as a proxy for learning difficulty in an ablation study, it's a simplified measure. More sophisticated difficulty metrics could be explored.
  • Model Dependence: The ablation study indicated that the effectiveness of simple distance-based criticality might vary between models, suggesting that the proposed combined criticality is more generally applicable.

Summary updated

Abstract

In safety-critical domains like automated driving (AD), errors by the object detector may endanger pedestrians and other vulnerable road users (VRU). As common evaluation metrics are not an adequate safety indicator, recent works employ approaches to identify safety-critical VRU and back-annotate the risk to the object detector. However, those approaches do not consider the safety factor in the deep neural network (DNN) training process. Thus, state-of-the-art DNN penalizes all misdetections equally irrespective of their criticality. Subsequently, to mitigate the occurrence of critical failure cases, i.e., false negatives, a safety-aware training strategy might be required to enhance the detection performance for critical pedestrians. In this paper, we propose a novel safety-aware loss variation that leverages the estimated per-pedestrian criticality scores during training. We exploit the reachability set-based time-to-collision (TTC-RSB) metric from the motion domain along with distance information to account for the worst-case threat quantifying the criticality. Our evaluation results using RetinaNet and FCOS on the nuScenes dataset demonstrate that training the models with our safety-aware loss function mitigates the misdetection of critical pedestrians without sacrificing performance for the general case, i.e., pedestrians outside the safety-critical zone.
BibTeX
@misc{lyssenko2024safety,
  author = {Maria Lyssenko and Piyush Pimplikar and Maarten Bieshaar and Farzad Nozarian and Rudolph Triebel},
  title = {{A Safety-Adapted Loss for Pedestrian Detection in Automated Driving}},
  year = {2024},
  eprint = {2402.02986},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.02986},
}

Loading related content...

Figure