Confident Object Detection via Conformal Prediction and Conformal Risk Control: an Application to Railway Signaling

Léo Andéol, Thomas Fel, Florence de Grancey, Luca Mossina

Introduction

Deep learning models are increasingly deployed in safety-critical environments, yet they often lack the ability to provide reliable confidence estimates. In railway signaling, where missing a "stop" signal can have catastrophic consequences, point predictions are insufficient. This paper addresses the challenge of providing trustworthy predictors for Traffic Light Localization (TLL) by wrapping "black-box" object detectors in a statistical safety blanket using Conformal Prediction (CP) and Conformal Risk Control (CRC).

Signal localization in a railway environment.

Methodology

The authors employ two primary frameworks to generate bounding boxes with formal guarantees:

  1. Split Conformal Prediction (CP): This method utilizes a calibration dataset to determine a "nonconformity score," representing how unusual a new data point is relative to the training distribution. It produces a prediction set—an enlarged bounding box—guaranteed to contain the ground truth with a user-defined probability (e.g., 90%).
  2. Conformal Risk Control (CRC): A generalization of CP, CRC allows for controlling more nuanced risks, such as the proportion of pixels missed or average recall across an image, rather than a simple binary coverage metric.

A key innovation presented is the use of max-additive and max-multiplicative scores. Traditional CP for object detection often relies on a conservative Bonferroni correction, which can lead to excessively large boxes. By calculating the quantile of the maximum residual across all coordinates (x, y, width, height) simultaneously, the authors achieve tighter, more practical "safety margins."

Dataset and Experimental Setup

The study introduces a novel railway signaling dataset comprising 3,414 images and 3,508 manually annotated bounding boxes. The data was sourced from 41 videos of French railway lines, capturing a wide variety of conditions:

  • Weather: Rain, snow, and clear skies.
  • Time of Day: Day and night scenarios.
  • Occlusion: Signals partially obscured by foliage or other structures.
A typical railway signal targeted for detection.

The framework was evaluated using three state-of-the-art object detectors: YOLOv5m, DETR-ResNet50, and DiffusionDet.

Results and Discussion

The experimental results confirm that both CP and CRC successfully produce empirical coverage close to target error rates. For instance, YOLOv5m achieved an average coverage of 0.93–0.95 against a target of 0.90.

Key insights include:

  • Model Agnosticism: The framework requires no retraining of the underlying "black-box" models, making it highly versatile.
  • Efficiency of Max-Scores: The "max-additive" approach consistently reduced the "stretch" (the ratio of the enlarged box to the original prediction), making the output more useful for human operators.
  • Threshold Effect: In image-wise conformalization, if a model fails to detect a signal, the conformal margin expands dramatically. This "explosion" serves as an inherent warning system for model failure.
  • CRC Advantages: Using CRC for pixel-wise recall resulted in the most reasonably sized bounding boxes while maintaining valid average guarantees.

Conclusion

This work demonstrates that conformal prediction frameworks can provide formal, distribution-free guarantees for object detection in safety-critical railway applications. While the methods rely on the assumption of data exchangeability, they offer a significant step toward deploying reliable AI in certified systems. Future research will explore extending these guarantees to instance segmentation and continuous video streams.

Summary updated

Abstract

Deploying deep learning models in real-world certi ed systems requires the ability to provide con dence estimates that accurately reflect their uncertainty. In this paper, we demonstrate the use of the conformal prediction framework to construct reliable and trustworthy predictors for detecting railway signals. Our approach is based on a novel dataset that includes images taken from the perspective of a train operator and state-of-the-art object detectors. We test several conformal approaches and introduce a new method based on conformal risk control. Our fi ndings demonstrate the potential of the conformal prediction framework to evaluate model performance and provide practical guidance for achieving formally guaranteed uncertainty bounds.
BibTeX
@inproceedings{andeol2023confident,
  author = {Léo Andéol and Thomas Fel and Florence de Grancey and Luca Mossina},
  title = {{Confident Object Detection via Conformal Prediction and Conformal Risk Control: an Application to Railway Signaling}},
  booktitle = {COPA},
  year = {2023},
  url = {https://openreview.net/forum?id=O4KHU6M6nS},
}
Figure