Revisiting Out-of-Distribution Detection in Real-time Object Detection: From Benchmark Pitfalls to a New Mitigation Paradigm

Changshun Wu, Weicheng He, Chih‐Hong Cheng, Xiaowei Huang, Saddek Bensalem

TL;DR

This paper reveals that popular Out-of-Distribution (OoD) benchmarks for object detection are fundamentally flawed, with up to 13% of "unknown" test objects actually belonging to known classes. To fix this, the authors introduce a new training-time paradigm that fine-tunes detectors on "proximal" OoD data, reducing hallucination errors by up to 91% while maintaining standard detection performance.

What Problem Does This Paper Address?

Deep learning models are notoriously overconfident. In object detection, when a model encounters an object it wasn't trained on (Out-of-Distribution or OoD), it often "hallucinates"—confidently labeling a penguin as a "car" or a "person."

While researchers have developed post-hoc filters to catch these errors, this paper identifies two major roadblocks:

  1. Benchmark Pitfalls: Existing evaluation datasets are messy. They contain "Type 1" errors (known objects labeled as unknown in test sets) and "Type 2" errors (unlabeled unknown objects in the training sets). This makes current performance metrics unreliable.
  2. Algorithmic Limits: Most current methods try to filter out OoD objects after the model has already made a prediction. The authors argue we should instead teach the model to never "see" these objects as foreground in the first place.

Methods

The authors propose a two-pronged approach:

  • Automated Data Curation: They built a pipeline using the FiftyOne library and the YOLOE open-vocabulary model to systematically clean datasets. This pipeline identifies and removes "ID contamination" (known classes) from OoD sets and vice versa.
  • Proximal OoD Fine-tuning: Instead of just training on known classes (ID), they fine-tune the model using "proximal" OoD samples—objects that are semantically similar to the training data but don't belong to any specific class (e.g., training on "cars" and "trucks," then using "tractors" as proximal OoD).
  • Objectness Suppression: During fine-tuning, the model is taught to treat these proximal OoD samples as "background." This forces the model to sharpen its "objectness" decision boundary, effectively learning to ignore things that look almost like known objects but aren't.

Datasets

The study utilizes several standard and modified benchmarks:

  • In-Distribution (ID): BDD-100K (driving scenes) and PASCAL-VOC (general objects).
  • Out-of-Distribution (OoD): OpenImages, Caltech-256, and Object365.
  • New Benchmarks: The authors introduced curated Near-OoD (visually similar to ID) and Far-OoD (distinctly different) datasets to allow for more granular evaluation.

Key Insights and Ideas

  • The Benchmark "Lie": The discovery that ~13% of objects in OoD test sets are actually ID classes is a wake-up call for the community. It means many "state-of-the-art" methods might be performing better (or worse) than we thought because the yardstick was crooked.
  • Objectness as a Shield: In object detection, the "objectness" score is the first gatekeeper. By suppressing this score for OoD-like objects during training, the authors prevent the classification head from even getting a chance to hallucinate.
  • Why Models Hallucinate: Using XAI (Saliency Maps), the authors show that vanilla models often hallucinate because they fixate on background textures (like green grass or water) rather than the object itself. Fine-tuning on proximal OoD teaches the model to ignore these spurious correlations and focus strictly on discriminative object features.

Key Results

  • Massive Error Reduction: The proposed fine-tuning led to a 91% reduction in hallucination errors for YOLO models on the BDD-100K dataset.
  • Architecture Agnostic: The method proved effective across different paradigms, including one-stage (YOLOv10), two-stage (Faster R-CNN), and Transformer-based (RT-DETR) detectors.
  • Minimal Performance Hit: Unlike many safety-oriented methods that tank the model's accuracy on known classes, this approach resulted in only a negligible (~1%) drop in Mean Average Precision (mAP).
  • Few-Shot Efficiency: The authors demonstrated that the model could be "fixed" with as few as 64–200 proximal OoD samples, making it practical for real-world deployment where data might be scarce.

Limitations and Future Work

  • Class Boundary Ambiguity: The authors acknowledge that "Visual Similarity at the Class Boundary" remains a challenge. For example, a model might still struggle to distinguish a white tiger cub (OoD) from a domestic cat (ID) due to extreme visual overlap.
  • Contextual Bias: While the method reduces reliance on background cues, it doesn't eliminate it entirely.
  • Future Directions: The authors suggest exploring hierarchical OoD definitions and more diverse synthetic OoD sources to further harden models against "near-miss" hallucinations.

Paper Link: arXiv:2503.07330
Code & Data: m-hood GitLab

Summary updated

Abstract

Out-of-distribution (OoD) inputs pose a persistent challenge to deep learning models, often triggering overconfident predictions on non-target objects. While prior work has primarily focused on refining scoring functions and adjusting test-time thresholds, such algorithmic improvements offer only incremental gains. We argue that a rethinking of the entire development lifecycle is needed to mitigate these risks effectively. This work addresses two overlooked dimensions of OoD detection in object detection. First, we reveal fundamental flaws in widely used evaluation benchmarks: contrary to their design intent, up to 13% of objects in the OoD test sets actually belong to in-distribution classes, and vice versa. These quality issues severely distort the reported performance of existing methods and contribute to their high false positive rates. Second, we introduce a novel training-time mitigation paradigm that operates independently of external OoD detectors. Instead of relying solely on post-hoc scoring, we fine-tune the detector using a carefully synthesized OoD dataset that semantically resembles in-distribution objects. This process shapes a defensive decision boundary by suppressing objectness on OoD objects, leading to a 91% reduction in hallucination error of a YOLO model on BDD-100K. Our methodology generalizes across detection paradigms such as YOLO, Faster R-CNN, and RT-DETR, and supports few-shot adaptation. Together, these contributions offer a principled and effective way to reduce OoD-induced hallucination in object detectors. Code and data are available at: https://gricad-gitlab.univ-grenoble-alpes.fr/dnn-safety/m-hood.
BibTeX
@inproceedings{wu2025revisiting,
  author = {Changshun Wu and Weicheng He and Chih‐Hong Cheng and Xiaowei Huang and Saddek Bensalem},
  title = {{Revisiting Out-of-Distribution Detection in Real-time Object Detection: From Benchmark Pitfalls to a New Mitigation Paradigm}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11245852},
}
Figure