Risk Ranked Recall: Collision Safety Metric for Object Detection Systems in Autonomous Vehicles

Ayoosh Bansal, Jayati Singh, Micaela Verucchi, Marco Caccamo, Lui ShaMECO 2021 · DOI · OpenReview

TL;DR

This paper introduces "Risk Ranked Recall" (R³), a new metric for evaluating object detection systems in autonomous vehicles. R³ goes beyond traditional recall by categorizing detected objects based on their collision risk, providing a more safety-aware assessment crucial for critical applications like autonomous driving.

What Problem Does This Paper Address?

Autonomous vehicles (AVs) rely heavily on object detection systems to perceive their surroundings and avoid collisions. While standard metrics like precision and recall are useful, they don't fully capture the safety implications of detection failures. For instance, failing to detect a car directly in front of the AV is far more dangerous than missing a car several lanes away. Existing metrics treat all false negatives equally, failing to prioritize detections that are critical for immediate safety. This paper addresses this gap by proposing a metric that explicitly ranks detected objects by their collision risk, allowing for a more nuanced and safety-focused evaluation of object detection systems.

Methods

The core contribution is the Risk Ranked Recall (R³) metric. This metric categorizes detected objects into three ranks based on an objective cyber-physical model of collision risk:

  • R³ (Imminent Collision): Objects that pose an immediate risk of collision if no action is taken.
  • R³ (Potential Collision): Objects that could potentially collide with the ego vehicle.
  • R³ (Other Objects): All other detected objects in the environment.

Recall is then calculated separately for each of these ranks. To determine these ranks, the authors define:

  • Time Horizon: The analysis is limited to the Time To Stop (TTS) of the ego vehicle, assuming an emergency brake.
  • True Positives: Instead of the standard Intersection over Union (IoU), they use Intersection over Groundtruth (IoG) to match predictions to ground truth. This is because for safety, it's more important that a predicted bounding box covers the actual object, even if it's slightly larger or overlaps with adjacent objects, rather than strictly adhering to a precise IoU threshold.
  • Collision Risk Model:
    • Imminent Collision: Determined by simulating trajectories of the ego vehicle and detected objects over the TTS horizon. If their bounding boxes overlap at any point, it's an imminent collision.
  • Potential Collision: Determined by calculating the "existence regions" of the ego vehicle and the object. If these regions overlap or their minimum distance falls below a critical threshold ($d_{crit}$) at any point within the TTS horizon, it's a potential collision.

Datasets

The authors use the Waymo Open Dataset [3] for evaluation. They specifically mention using 24 sequences from this dataset. They also note the potential need for synthetic datasets that can include imminently unsafe situations, as real-world datasets may not sufficiently capture such scenarios.

Key Insights and Ideas

  • Safety-Awareness is Crucial: Traditional metrics like recall are insufficient for safety-critical applications like AVs because they don't differentiate the severity of detection failures.
  • Risk-Based Prioritization: Ranking objects by their collision risk allows for a more meaningful evaluation of an object detection system's ability to prevent accidents.
  • IoG vs. IoU: Using IoG for matching predictions to ground truth is a more robust approach for safety-critical scenarios, as it prioritizes covering the object over precise bounding box alignment.
  • Imminent vs. Potential Collisions: Distinguishing between these two types of risks provides a finer-grained understanding of system performance.
  • Resolution Matters, but R³ is Robust: While input image resolution affects recall, the R³ metric shows less variation, suggesting it's a more stable indicator of safety performance across different system configurations.

Key Results

The paper demonstrates the R³ metric using pre-trained object detection networks (YOLOv3, Faster R-CNN, Mask R-CNN) on the Waymo Open Dataset. They found that:

  • Out of 27,973 labeled objects, 952 were identified to pose a Potential risk of collision.
  • No objects were identified to pose a risk of Imminent collision in the Waymo dataset, highlighting the need for synthetic data to evaluate this critical rank.
  • Figure 5 shows plots of R³ and Recall versus confidence for various networks and resolutions. These plots visually demonstrate how R³ provides a different perspective on performance compared to standard recall, particularly emphasizing safety-critical false negatives.

Limitations and Future Work

  • Lack of Imminent Collisions in Datasets: The authors acknowledge that real-world datasets like Waymo may not contain enough instances of imminent collisions, limiting the evaluation of the R³ (Imminent Collision) rank. They suggest the need for synthetic datasets to address this.
  • Metric Integration: Future work should focus on establishing the metric's efficacy in influencing design decisions for perception systems, including sensor fusion and tracking.
  • Loss Function Redesign: The metrics could be used to redesign loss functions for training object detection DNNs, which would need to consider both precision and recall.
  • Precision Component: The current R³ metric focuses solely on recall. Future work could explore incorporating precision into a risk-ranked framework.

Summary updated

Abstract

Commonly used metrics for evaluation of object detection systems (precision, recall, mAP) do not give complete information about their suitability of use in safety critical tasks, like obstacle detection for collision avoidance in Autonomous Vehicles (AV). This work introduces the Risk Ranked Recall ($R^3$) metrics for object detection systems. The $R^3$ metrics categorize objects within three ranks. Ranks are assigned based on an objective cyber-physical model for the risk of collision. Recall is measured for each rank.
BibTeX
@inproceedings{bansal2021risk,
  author = {Ayoosh Bansal and Jayati Singh and Micaela Verucchi and Marco Caccamo and Lui Sha},
  title = {{Risk Ranked Recall: Collision Safety Metric for Object Detection Systems in Autonomous Vehicles}},
  booktitle = {MECO},
  year = {2021},
  doi = {10.1109/meco52532.2021.9460196},
}
Figure