TL;DR
This paper moves beyond standard object detection metrics like mAP to address the real-world needs of fire alarm systems: making a reliable binary decision (alarm or no alarm) for every image. By applying a split conformal procedure, the author demonstrates how to calibrate confidence thresholds to guarantee a specific maximum miss rate, showing that standard fixed thresholds are dangerously inconsistent across different models.
What Problem Does This Paper Address?
In research, compact object detectors like YOLOv8n are typically evaluated using box-level metrics such as mean Average Precision (mAP). However, a deployed fire monitoring system doesn't care about the exact coordinates of a bounding box as much as it cares about a single question: "Is there a fire in this frame?"
Currently, most systems use an arbitrary fixed confidence threshold (e.g., 0.25 or 0.5) to trigger an alarm. The author points out two major flaws in this approach:
- Model Inconsistency: Detector confidence scores are not calibrated probabilities. A threshold of 0.5 might result in a 17% miss rate for one model and an 8% miss rate for another.
- Lack of Control: A fixed threshold gives engineers no way to guarantee a specific safety level, such as "miss no more than 1% of fires."
Methods
The paper evaluates two compact detectors—YOLOv8n and YOLOv5s—on the D-Fire dataset. The core of the approach involves two steps:
1. From Boxes to an Image Score
Since detectors produce multiple boxes per image, the author compares three rules to consolidate these into a single "alarm score" $s(I)$:
- Max: The score is simply the confidence of the most certain box.
- Noisy-OR: A rule from multiple instance learning that allows several weak detections to "add up" to a higher alarm score.
- Area-gated: Similar to Max, but ignores boxes covering less than 0.1% of the image to reduce noise.
2. Calibrated Threshold Selection
Instead of picking a threshold by hand, the author uses a split conformal procedure. By looking at the scores a model gives to a "calibration set" of known positive images, they can mathematically determine a threshold $\hat{t}$ that ensures the probability of missing a new fire image is below a user-defined target $\alpha$ (e.g., 5%).
Datasets
The study uses the D-Fire dataset, which contains 21,527 images categorized into: fire only, smoke only, fire and smoke, and "none" (background images that might look like fire, such as sunsets or clouds). The evaluation was performed by repeatedly splitting the test set into calibration and evaluation halves to ensure the results were statistically robust.
Key Insights and Ideas
- mAP is Deceptive: While the models had relatively modest mAP scores (~0.75), their image-level performance was excellent, with Area Under the ROC Curve (AUC) values above 0.99. This is because "errors" like slightly misplaced boxes or missing a second fire in the same frame don't actually break the alarm system.
- Fixed Thresholds are Meaningless: The paper highlights that using a threshold of 0.5 led to a twofold difference in miss rates between the two YOLO versions. This proves that you cannot simply "copy-paste" settings between different model architectures.
- Noisy-OR Wins at Low False Alarms: The Noisy-OR rule generally provided the best trade-off, maintaining the lowest false alarm rates when trying to hit strict detection targets.
- Smoke is the "Hard" Class: Even when the overall miss rate target was met, the system was more likely to miss smoke-only images compared to fire-only images.

Key Results
- Target Accuracy: The calibrated thresholds successfully met miss-rate targets of 10%, 5%, and 1% for both detectors across all scoring rules.
- False Alarm Trade-off: At a 5% miss target, the false alarm rate stayed impressively low (between 1.2% and 3.0%). However, pushing for a 1% miss target caused false alarms to spike significantly (up to 21.9% for some rules), as the threshold must drop to near zero to catch the hardest cases.
- Category Performance: As shown in Figure 2, images containing both fire and smoke were detected almost 100% of the time, while smoke-only images were the primary source of misses.

Limitations and Future Work
- Single-Frame Analysis: The current work treats each image independently. Real-world systems process video, where temporal information (e.g., "is this smoke plume growing?") could significantly reduce false alarms.
- Dataset Specificity: The calibration assumes that new images will be "exchangeable" (statistically similar) to the calibration set. If a camera is moved to a radically different environment, the calibration might fail.
- Failure Cases: The author notes that some misses are due to extremely small, distant fires or ambiguous labels (like steam from cooling towers labeled as smoke).

Summary updated