From Evaluation to Verification: Towards Task-Oriented Relevance Metrics for Pedestrian Detection in Safety-Critical Domains

Maria Lyssenko, Christoph Gladisch, Christian Heinzemann, Matthias Woehrle, Rudolph TriebelWorkshops @ CVPR 2021 · OpenReview

Abstract

Whenever a visual perception system is employed in safety-critical applications such as automated driving, a thorough, task-oriented experimental evaluation is necessary to guarantee safe system behavior. While most standard evaluation methods in computer vision provide a good comparability on benchmarks, they tend to fall short on assessing the system performance that is actually relevant for the given task. In our work, we consider pedestrian detection as a highly relevant perception task, and we argue that standard measures such as Intersection over Union (IoU) give insufficient results, mainly because they are insensitive to important physical cues including distance, speed, and direction of motion. Therefore, we investigate so-called relevance metrics, where specific domain knowledge is exploited to obtain a task-oriented performance measure focusing on distance in this initial work. Our experimental setup is based on the CARLA simulator and allows a controlled evaluation of the impact of that domain knowledge. Our first results indicate a linear decrease of the IoU related to the pedestrians' distance, leading to the proposal of a first relevance metric that is also conditioned on the distance.

Introduction

Standard evaluation methods in computer vision, such as mean Average Precision (mAP) or Intersection over Union (IoU), provide excellent comparability across benchmarks but often fail to assess performance in a task-oriented manner. In safety-critical domains like automated driving, these metrics are "task-agnostic"—they do not distinguish between a missed pedestrian at five meters versus fifty meters. This paper argues that such metrics are insufficient for safety verification and proposes "relevance metrics" that incorporate domain-specific knowledge, specifically physical distance, to better evaluate system safety.

Methodology

The study utilizes a controlled experimental pipeline based on the CARLA simulator (Town03 map). Pedestrians were spawned at distances ranging from 2 to 120 meters with varying postures. A DeepLabv3+ network with a ResNet-101 backbone was trained for semantic segmentation. To enable distance-aware evaluation, the researchers fused three data streams: RGB images for model processing, depth maps for precise distance ground truth, and semantic ground truth for accuracy assessment.

A key contribution is the introduction of the $dIoU_\delta$ metric, which defines the maximum distance ($d$) at which a model maintains a specific minimum IoU threshold ($\delta$). This shifts the focus from average performance to worst-case guarantees.

Analysis of Distance-Dependent Performance

A primary finding of this research is the fundamental relationship between distance and detection quality. Experimental results demonstrate that IoU decreases linearly as the distance to the pedestrian increases.

Mean IoU and quantiles as a function of pedestrian distance, showing a clear linear decay trend.

This linear decay suggests that model performance is not just a matter of object size but is deeply tied to how perception networks handle scale and depth. By analyzing quantiles (e.g., 0.2 and 0.8), the researchers can identify the variance in performance at different ranges, providing a more nuanced view than a single mean value.

Results and Safety Verification

The model achieved an overall accuracy of 84.8% and a mean IoU (mIoU) of 53.9% on the full-resolution test set. However, the task-oriented analysis provides more actionable safety insights. By applying the $dIoU_\delta$ metric, the authors established "guaranteed detection distances" for different safety requirements.

Visualization of the proposed relevance metric, identifying maximum distances for specific IoU thresholds (Area 1: 0.5 IoU, Area 2: 0.15 IoU).

As shown in the figure above:

  • To guarantee a high-quality detection (IoU $\ge$ 0.5), the maximum safe distance is 33 meters.
  • To guarantee at least a partial detection (IoU $\ge$ 0.15), the distance extends to 54 meters.

Beyond these distances, the model's performance becomes too inconsistent for safety guarantees, even if the average IoU remains relatively high.

Conclusion and Future Work

This work demonstrates that physical distance is a powerful proxy for relevance in pedestrian detection. By moving from statistical averages to distance-conditioned verification, developers can better define the operational design domain (ODD) of a perception system.

Future research will address the "simulation gap" by validating these findings on real-world data. Additionally, the authors plan to expand the relevance model to include dynamic factors such as Time-To-Collision (TTC), pedestrian velocity, and orientation, and transition from semantic to instance segmentation to better handle overlapping objects.

Summary updated

Abstract

Whenever a visual perception system is employed in safety-critical applications such as automated driving, a thorough, task-oriented experimental evaluation is necessary to guarantee safe system behavior. While most standard evaluation methods in computer vision provide a good comparability on benchmarks, they tend to fall short on assessing the system performance that is actually relevant for the given task. In our work, we consider pedestrian detection as a highly relevant perception task, and we argue that standard measures such as Intersection over Union (IoU) give insufficient results, mainly because they are insensitive to important physical cues including distance, speed, and direction of motion. Therefore, we investigate so-called relevance metrics, where specific domain knowledge is exploited to obtain a task-oriented performance measure focusing on distance in this initial work. Our experimental setup is based on the CARLA simulator and allows a controlled evaluation of the impact of that domain knowledge. Our first results indicate a linear decrease of the IoU related to the pedestrians' distance, leading to the proposal of a first relevance metric that is also conditioned on the distance.
BibTeX
@inproceedings{lyssenko2021from,
  author = {Maria Lyssenko and Christoph Gladisch and Christian Heinzemann and Matthias Woehrle and Rudolph Triebel},
  title = {{From Evaluation to Verification: Towards Task-Oriented Relevance Metrics for Pedestrian Detection in Safety-Critical Domains}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=MRPuOH8Fsy},
}
Figure