Introduction
Deep learning models are increasingly deployed in safety-critical environments, yet they often lack the ability to provide reliable confidence estimates. In railway signaling, where missing a "stop" signal can have catastrophic consequences, point predictions are insufficient. This paper addresses the challenge of providing trustworthy predictors for Traffic Light Localization (TLL) by wrapping "black-box" object detectors in a statistical safety blanket using Conformal Prediction (CP) and Conformal Risk Control (CRC).
Methodology
The authors employ two primary frameworks to generate bounding boxes with formal guarantees:
- Split Conformal Prediction (CP): This method utilizes a calibration dataset to determine a "nonconformity score," representing how unusual a new data point is relative to the training distribution. It produces a prediction set—an enlarged bounding box—guaranteed to contain the ground truth with a user-defined probability (e.g., 90%).
- Conformal Risk Control (CRC): A generalization of CP, CRC allows for controlling more nuanced risks, such as the proportion of pixels missed or average recall across an image, rather than a simple binary coverage metric.
A key innovation presented is the use of max-additive and max-multiplicative scores. Traditional CP for object detection often relies on a conservative Bonferroni correction, which can lead to excessively large boxes. By calculating the quantile of the maximum residual across all coordinates (x, y, width, height) simultaneously, the authors achieve tighter, more practical "safety margins."
Dataset and Experimental Setup
The study introduces a novel railway signaling dataset comprising 3,414 images and 3,508 manually annotated bounding boxes. The data was sourced from 41 videos of French railway lines, capturing a wide variety of conditions:
- Weather: Rain, snow, and clear skies.
- Time of Day: Day and night scenarios.
- Occlusion: Signals partially obscured by foliage or other structures.
The framework was evaluated using three state-of-the-art object detectors: YOLOv5m, DETR-ResNet50, and DiffusionDet.
Results and Discussion
The experimental results confirm that both CP and CRC successfully produce empirical coverage close to target error rates. For instance, YOLOv5m achieved an average coverage of 0.93–0.95 against a target of 0.90.
Key insights include:
- Model Agnosticism: The framework requires no retraining of the underlying "black-box" models, making it highly versatile.
- Efficiency of Max-Scores: The "max-additive" approach consistently reduced the "stretch" (the ratio of the enlarged box to the original prediction), making the output more useful for human operators.
- Threshold Effect: In image-wise conformalization, if a model fails to detect a signal, the conformal margin expands dramatically. This "explosion" serves as an inherent warning system for model failure.
- CRC Advantages: Using CRC for pixel-wise recall resulted in the most reasonably sized bounding boxes while maintaining valid average guarantees.
Conclusion
This work demonstrates that conformal prediction frameworks can provide formal, distribution-free guarantees for object detection in safety-critical railway applications. While the methods rely on the assumption of data exchangeability, they offer a significant step toward deploying reliable AI in certified systems. Future research will explore extending these guarantees to instance segmentation and continuous video streams.
Summary updated