Confident Object Detection via Conformal Prediction and Conformal Risk Control: an Application to Railway Signaling

Léo Andéol, Thomas Fel, Florence De Grancey, Luca Mossina

Introduction

Deep learning models have revolutionized computer vision, yet their deployment in safety-critical environments—such as autonomous railway signaling—remains a challenge. Standard object detection models are often "overconfident," providing predictions without a reliable measure of their own uncertainty. In the context of railway safety, a missed signal or a mislocated traffic light is not merely a software error; it is a potential catastrophe.

To bridge the gap between high-performance deep learning and the rigorous requirements of certified safety systems, this paper explores the use of Conformal Prediction (CP) and Conformal Risk Control (CRC). These frameworks allow researchers to transform standard "black-box" detectors into trustworthy systems that provide statistically sound safety margins, ensuring that predicted search areas contain the ground truth with a user-defined level of confidence.

The Problem: Uncertainty in Object Detection

While simple classification models can output a probability score for a single label, Object Detection (OD) is significantly more complex. It involves predicting both the coordinates (localization) and the classes for a variable number of objects within an image. Traditional methods for Uncertainty Quantification (UQ) often require retraining models or making restrictive assumptions about data distribution.

This research seeks a distribution-free, model-agnostic approach. The goal is to provide a formal guarantee that a predicted bounding box—expanded by a calculated safety margin—will encompass the actual object, regardless of the underlying model architecture or the complexity of the scene.

Methodology: Conformalizing the Detector

The authors employ two primary frameworks to "conformalize" existing detectors using a held-out calibration dataset:

  1. Split Conformal Prediction (CP): This method calculates a "nonconformity score" based on how much the model's predictions typically deviate from the ground truth in the calibration set. During inference, the system adds a safety margin to the bounding boxes to ensure the ground truth is captured with a specific probability (e.g., 90%).
  2. Conformal Risk Control (CRC): A more flexible generalization of CP. Rather than a binary "is the object inside or out?" metric, CRC allows for controlling a continuous risk, such as pixel-wise recall. This ensures that, on average, a specific percentage of target pixels are covered, preventing the "exploded" bounding boxes that can occur with traditional CP.

Technical Innovations

To improve these frameworks, the authors introduced Max-Error Scores (max-additive and max-multiplicative). These scores improve upon the standard Bonferroni correction—which is often overly conservative—by looking at the maximum residual across all four box coordinates simultaneously. Furthermore, they compared box-wise guarantees (for individual detections) against image-wise guarantees, which better account for false negatives across an entire frame.

The Railway Traffic Light Detection (RTLD) Dataset

To test these methods in a realistic setting, the authors introduced a novel, purpose-built dataset for Railway Traffic Light Detection (RTLD). Sourced from 41 videos of French railway lines, the dataset includes 3,414 images with 3,508 manually annotated bounding boxes. The data captures a wide array of challenging conditions, including rain, snow, night-time operations, and foliage occlusion, providing a robust benchmark for safety-critical UQ.

Key Insights and Results

The framework was tested on three state-of-the-art detectors: YOLOv5m, DETR-ResNet50, and DiffusionDet. Several key findings emerged:

  • The Threshold Effect: Traditional CP (using Hausdorff distance) can be brittle. If a model misses a single object during calibration, the safety margin required to satisfy the guarantee can expand to cover the entire screen.
  • CRC as a Practical Solution: By utilizing Conformal Risk Control (specifically pixel-wise recall), the system maintained formal guarantees while producing much more usable bounding boxes. For the DiffusionDet model, CRC achieved the desired risk level with a "stretch factor" (the ratio of the conformal box area to the predicted box area) of only 1.56x, compared to a massive 9.83x required by traditional CP.
  • Model Agnosticism: The conformal wrapper provided consistent statistical safety regardless of whether the underlying architecture was Transformer-based (DETR) or convolution-based (YOLO).
  • Post-Hoc Efficiency: A major advantage of this approach is that it is performed after the model is trained. It requires no expensive retraining, making it a computationally lightweight addition to existing deployment pipelines.

Conclusion and Future Directions

This research demonstrates that conformal frameworks can provide the formal uncertainty bounds necessary for industrial certification in railway signaling. While the quality of the conformal margins is ultimately limited by the quality of the underlying detector, the move toward Conformal Risk Control offers a viable path for practical implementation.

Future work will focus on scaling the dataset to achieve the ultra-low risk levels required for full safety certification and extending the framework from bounding boxes to instance segmentation. Moving to pixel-level masks would allow CRC’s pixel-wise guarantees to be even more precise, further reducing the safety margins while maintaining rigorous reliability.

Summary updated

Abstract

Deploying deep learning models in real-world certified systems requires the ability to provide confidence estimates that accurately reflect their uncertainty. In this paper, we demonstrate the use of the conformal prediction framework to construct reliable and trustworthy predictors for detecting railway signals. Our approach is based on a novel dataset that includes images taken from the perspective of a train operator and state-of-the-art object detectors. We test several conformal approaches and introduce a new method based on conformal risk control. Our findings demonstrate the potential of the conformal prediction framework to evaluate model performance and provide practical guidance for achieving formally guaranteed uncertainty bounds.
BibTeX
@inproceedings{andeol2023confident,
  author = {Léo Andéol and Thomas Fel and Florence De Grancey and Luca Mossina},
  title = {{Confident Object Detection via Conformal Prediction and Conformal Risk Control: an Application to Railway Signaling}},
  booktitle = {COPA},
  year = {2023},
  url = {https://openreview.net/forum?id=O4KHU6M6nS},
}
Figure