Fast object detectors such as YOLO find and label objects in a single pass over the image. They are what gets deployed on drones, vehicles and factory cameras. They also have a failure that no score threshold fixes: they "see" objects in empty background with very high confidence. From COCO to COCO-FP built a set of 3,772 images that contain none of the 80 COCO categories. Detectors including YOLOv9 still fire on these images at confidence above 0.9, and adding the images to the COCO validation set drops YOLOv9-E's AP50 from 72.8 to 65.7. Selective prediction, meaning models that may say "I don't know" and leave an input unanswered, offers formal tools for bounding errors on the outputs a model chooses to keep. The detection versions of these tools so far bound the wrong side of the problem. They guarantee that true objects are not missed, and say nothing about how many false alarms are emitted.
We propose to give a single-stage detector a user-set, statistically guaranteed budget on false alarms per image, for example "on average at most 0.1 spurious boxes per frame." Two pieces make this work. The first is a small trained rejection head that scores each candidate box on whether it will turn out to match no real object. This score is separate from the class score. The second is a conformal calibration step that sets the rejection threshold so the expected number of false alarms per image stays below the budget. The guarantee comes from the calibration step and holds for any score. The rejection head matters because a better score keeps more true objects at the same budget. The research question is whether a dedicated head buys recall that simpler fixes cannot.
Core Claim
The expected count of unmatched boxes in an image can only fall as the rejection threshold rises. That makes it a monotone loss, so Conformal Risk Control applies to it directly. That paper calibrates a threshold on a held-out set so the expected test loss stays below a target, for any monotone loss, with a slack of order $1/n$ in the calibration size. Exchangeability is needed only at the level of whole images, so the dependence between boxes within an image does not break the guarantee.
Define the per-image loss as
$$L_i(\lambda) = \#\{\text{emitted boxes in image } i \text{ with selection score} \ge \lambda \text{ that match no ground-truth object}\},$$
where $\lambda$ is the rejection threshold. Calibration picks the smallest $\hat\lambda$ that satisfies $\mathbb{E}[L(\hat\lambda)] \le \alpha$ on held-out images, where $\alpha$ is the false-alarm budget. The guarantee is cheap. What is not cheap is recall at a fixed budget, and recall is fully determined by how well the score separates real objects from hallucinated ones. COCO-FP shows that the class confidence ranks these poorly. Our claim is that a rejection head trained specifically for this separation recovers substantially more true objects under the same certified budget.
Relation to Prior Work
The closest work is Conformal Object Detection by Sequential Risk Control (SeqCRC). It wraps a frozen YOLOv8 or DETR, calibrates a confidence threshold, widens box margins, and outputs sets of possible labels, all so that true objects are covered at a target rate. Its guarantees concern recall. As the file summarizes, it does not control precision, and it could meet its guarantee with overly broad outputs.
The earlier railway application, Confident Object Detection via Conformal Prediction and Conformal Risk Control, calibrates joint box margins for a single class using YOLOv5m. It reports empirical coverage of 0.93–0.95 against a 0.90 target, and it observes that margins blow up when the detector misses an object. Both papers are post-hoc and frozen-model, and both control misses. We control the opposite error, and we train the score that is fed to the calibration step. The two kinds of guarantee are complementary, and we test whether they can be combined.
SelectiveNet supplies our method. It adds a learned selection head to a classifier, plus an auxiliary head that keeps the shared features from collapsing onto the accepted subset. It reports up to 48% relative risk improvement over softmax-confidence thresholding at 80% coverage on Cats vs. Dogs. We carry the selection head into a dense per-location detection head. Where SelectiveNet targets one coverage level, we hand the coverage choice to calibration time, which removes its need to retrain for each target. Selective Classification for Deep Neural Networks constrains our expectations: plain softmax confidence was often as good as Monte Carlo dropout for ranking, so a learned head has to beat a strong cheap baseline, not a weak one.
The COCO-FP study motivates the work and supplies the stress test. It offers no fix of its own. Inferring Spatial Uncertainty in Object Detection models box coordinates as random variables for LiDAR detection. It reminds us that localization error and hallucination are different failures. Our loss counts only boxes that match nothing at a standard overlap threshold, so it targets hallucination rather than sloppy boxes.
Research Program
We add to a YOLOv8 head a one-layer selection branch at every grid location. Its added cost should be well under 1% of the detector's floating-point operations, and we will measure frames per second to confirm. For each candidate that survives non-maximum suppression, the target is binary: whether the candidate is matched to a ground-truth box at IoU 0.5. Training data are COCO train images mixed with a pool of object-free ImageNet images. These must be disjoint from the COCO-FP test images; following COCO-FP's construction, we filter with a strong detector plus a human spot check. The original classification and box-regression losses stay on, playing the role of SelectiveNet's auxiliary head, so the detector's ordinary accuracy should not degrade. We also train a variant with a frozen backbone, which isolates the contribution of feature adaptation.
At calibration time we sweep $\lambda$ on held-out images and pick $\hat\lambda$ by conformal risk control for the false-alarm count. We also implement a two-constraint version. It first applies the SeqCRC confidence step to guarantee object retention, then calibrates false alarms on the retained set. We report whether both constraints can be met at once, or whether the budgets conflict and leave no feasible threshold. That second result is informative in its own right.
The most serious threat to validity is that the share of empty-background frames at deployment may differ from the share in calibration, which breaks exchangeability. We therefore calibrate on mixtures with a fixed background fraction. At test time we vary that fraction from 0% to 50% and measure how far realized false alarms overshoot the budget.
Baselines
The naive reference is YOLOv8 with its default confidence cutoff and no calibration. We report its false alarms per image and recall on COCO val mixed with COCO-FP. For scale, the file's only reported number on this mixture is the YOLOv9-E AP50 drop from 72.8 to 65.7 in COCO-FP. We also run YOLOv9-E so that number can be reproduced.
The strongest existing competitor is conformal risk control on the detector's own class confidence. This is the identical calibration procedure with the stock score substituted, on the same calibration images. It is the strongest competitor because it already delivers the guarantee, and our only advantage can be recall. Beating it means higher recall at matched certified budgets α ∈ {0.05, 0.1, 0.3, 1.0}, outside the run-to-run spread of three seeds.
We also report SeqCRC as published, which certifies recall, measuring the false alarms it emits at its own operating point. The file gives no reported numeric precision or mAP for SeqCRC on COCO ("no reported result found on Lacuna" beyond the statement that test risk stayed below α), so we rerun it with the same 2,500/2,500 calibration and test split it describes.
Two ablations isolate the key ingredient. The first is a hard-negative fine-tune: the same detector and the same background images, with background treated only as negatives for the classification head, no selection head, and an identical number of training steps and tuning budget. The second is a frozen-backbone selection head, which separates what the new score itself contributes from what changing the features contributes.
Why not just…?
Why not just raise the confidence threshold? A raised threshold carries no guarantee. The direct evidence against it is COCO-FP: hallucinations sit above 0.9 confidence, so any cutoff that removes them also removes many real objects. Our calibrated stock-score baseline is exactly "raise the threshold, but choose it rigorously." If it matches the learned head on recall, we will report that the guarantee alone is the contribution.
Why not just fine-tune the detector on background images? This is a real threat. Hard-negative training may fix much of the COCO-FP failure by itself. The matched hard-negative ablation settles it. If that ablation, run through the same calibration, matches the selection head's recall at each budget, the separate head is unnecessary and we drop it. The SelectiveNet result suggests a dedicated head plus a full-data auxiliary loss can beat folding rejection into the main output, but that evidence comes from classification, not detection.
Why not just gate whole images with an "any object present?" classifier? On COCO-FP, which is entirely empty images, an image-level gate could look nearly perfect. That is exactly why the test set must also count false alarms inside cluttered COCO images that do contain objects. We train this gate on the same data and calibrate it the same way. We report it separately on the empty-image portion and the object-containing portion of the test set, expecting it to fail on the latter.
Why not use Monte Carlo dropout or ensembles? They require many forward passes per frame, which defeats the purpose of a real-time detector. Selective Classification for Deep Neural Networks also found softmax confidence competitive with dropout for ranking. We include a five-member ensemble once, as an upper reference at roughly five times the inference cost.
Decisive Checks
The work succeeds if four things hold. Realized false alarms per image on held-out COCO plus COCO-FP stay at or below α across all budgets and seeds when background prevalence matches calibration. The learned score gives higher recall than the calibrated stock score at the same certified budget. It also beats the matched hard-negative fine-tune. And frame rate drops by less than 2%.
The prevalence sweep then tells practitioners how much safety margin to add when the share of empty frames is unknown. The two-constraint experiment shows whether miss guarantees and false-alarm guarantees can coexist on a single fast detector. If the learned head fails to beat the hard-negative fine-tune, the honest finding is that calibrated thresholding on a better-trained detector is enough.
References
- From COCO to COCO-FP: A Deep Dive into Background False Positives for COCO Detectors
- Conformal Risk Control
- Conformal Object Detection by Sequential Risk Control
- Confident Object Detection via Conformal Prediction and Conformal Risk Control: an Application to Railway Signaling
- SelectiveNet: A Deep Neural Network with an Integrated Reject Option
- Selective Classification for Deep Neural Networks
- Inferring Spatial Uncertainty in Object Detection