Calibrated Alarm Thresholds for Compact Fire and Smoke Detectors

Vikas SandireddyarXiv 2026 · arXiv

TL;DR

This paper moves beyond standard object detection metrics like mAP to address the real-world needs of fire alarm systems: making a reliable binary decision (alarm or no alarm) for every image. By applying a split conformal procedure, the author demonstrates how to calibrate confidence thresholds to guarantee a specific maximum miss rate, showing that standard fixed thresholds are dangerously inconsistent across different models.

What Problem Does This Paper Address?

In research, compact object detectors like YOLOv8n are typically evaluated using box-level metrics such as mean Average Precision (mAP). However, a deployed fire monitoring system doesn't care about the exact coordinates of a bounding box as much as it cares about a single question: "Is there a fire in this frame?"

Currently, most systems use an arbitrary fixed confidence threshold (e.g., 0.25 or 0.5) to trigger an alarm. The author points out two major flaws in this approach:

  1. Model Inconsistency: Detector confidence scores are not calibrated probabilities. A threshold of 0.5 might result in a 17% miss rate for one model and an 8% miss rate for another.
  2. Lack of Control: A fixed threshold gives engineers no way to guarantee a specific safety level, such as "miss no more than 1% of fires."

Methods

The paper evaluates two compact detectors—YOLOv8n and YOLOv5s—on the D-Fire dataset. The core of the approach involves two steps:

1. From Boxes to an Image Score

Since detectors produce multiple boxes per image, the author compares three rules to consolidate these into a single "alarm score" $s(I)$:

  • Max: The score is simply the confidence of the most certain box.
  • Noisy-OR: A rule from multiple instance learning that allows several weak detections to "add up" to a higher alarm score.
  • Area-gated: Similar to Max, but ignores boxes covering less than 0.1% of the image to reduce noise.

2. Calibrated Threshold Selection

Instead of picking a threshold by hand, the author uses a split conformal procedure. By looking at the scores a model gives to a "calibration set" of known positive images, they can mathematically determine a threshold $\hat{t}$ that ensures the probability of missing a new fire image is below a user-defined target $\alpha$ (e.g., 5%).

Datasets

The study uses the D-Fire dataset, which contains 21,527 images categorized into: fire only, smoke only, fire and smoke, and "none" (background images that might look like fire, such as sunsets or clouds). The evaluation was performed by repeatedly splitting the test set into calibration and evaluation halves to ensure the results were statistically robust.

Key Insights and Ideas

  • mAP is Deceptive: While the models had relatively modest mAP scores (~0.75), their image-level performance was excellent, with Area Under the ROC Curve (AUC) values above 0.99. This is because "errors" like slightly misplaced boxes or missing a second fire in the same frame don't actually break the alarm system.
  • Fixed Thresholds are Meaningless: The paper highlights that using a threshold of 0.5 led to a twofold difference in miss rates between the two YOLO versions. This proves that you cannot simply "copy-paste" settings between different model architectures.
  • Noisy-OR Wins at Low False Alarms: The Noisy-OR rule generally provided the best trade-off, maintaining the lowest false alarm rates when trying to hit strict detection targets.
  • Smoke is the "Hard" Class: Even when the overall miss rate target was met, the system was more likely to miss smoke-only images compared to fire-only images.

![Fig.

  1. Image-level ROC curves for the three rules. The horizontal axis is on a log scale to show the region of low false alarm rates.](/figures/art_2000d4725d7c44159c0511196f8b3083)

Key Results

  • Target Accuracy: The calibrated thresholds successfully met miss-rate targets of 10%, 5%, and 1% for both detectors across all scoring rules.
  • False Alarm Trade-off: At a 5% miss target, the false alarm rate stayed impressively low (between 1.2% and 3.0%). However, pushing for a 1% miss target caused false alarms to spike significantly (up to 21.9% for some rules), as the threshold must drop to near zero to catch the hardest cases.
  • Category Performance: As shown in Figure 2, images containing both fire and smoke were detected almost 100% of the time, while smoke-only images were the primary source of misses.

![Fig.

  1. Alarm rate per D-Fire category with the max rule at a 5% miss target. For the first three categories this is the detection rate; for “none” it is the false alarm rate.](/figures/art_9119bc6bbece44c9be5f291cc7f17050)

Limitations and Future Work

  • Single-Frame Analysis: The current work treats each image independently. Real-world systems process video, where temporal information (e.g., "is this smoke plume growing?") could significantly reduce false alarms.
  • Dataset Specificity: The calibration assumes that new images will be "exchangeable" (statistically similar) to the calibration set. If a camera is moved to a radically different environment, the calibration might fail.
  • Failure Cases: The author notes that some misses are due to extremely small, distant fires or ambiguous labels (like steam from cooling towers labeled as smoke).

![Fig.

  1. Failure cases of YOLOv8n. Top: false alarms (clouds, lamps). Bottom: misses (tiny distant fires, steam).](/figures/art_db68e587d79a44c98dbe1c3bba7a2d2f)

Summary updated

Abstract

Compact object detectors such as YOLOv8n are a common choice for camera-based fire and smoke detection on low-power devices. These detectors are usually evaluated with box-level metrics such as mean average precision, and their outputs are turned into decisions with a fixed confidence threshold such as 0.25 or 0.5. A fire monitoring system, however, makes one decision per image: raise an alarm or not. In this paper we evaluate two publicly available compact detectors on the D-Fire test set at the image level. We compare three simple rules for turning box detections into one alarm score per image, and we use a split conformal procedure to choose the alarm threshold from held-out images so that the expected fraction of missed fire and smoke images stays below a chosen target. We find that the same fixed threshold leads to very different miss rates for the two detectors, 17.3 and 8.4 percent at a threshold of 0.5, while the calibrated threshold meets miss-rate targets of 10, 5 and 1 percent for both detectors and all three rules. At a 5 percent target, the false alarm rate on images without fire or smoke stays between 1.2 and 3.0 percent. We also find that smoke-only images stay below the target even when the overall target is met, and we describe the main failure cases.
BibTeX
@misc{sandireddy2026calibrated,
  author = {Vikas Sandireddy},
  title = {{Calibrated Alarm Thresholds for Compact Fire and Smoke Detectors}},
  year = {2026},
  eprint = {2611.11757},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11757},
}
Figure