More Practical Scenario of Open-set Object Detection: Open at Category Level and Closed at Super-category Level

Yusuke Hosoya, Masanori Suganuma, Takayuki OkataniarXiv 2022 · arXiv · OpenReview

Abstract

Open-set object detection (OSOD) has recently attracted considerable attention. It is to detect unknown objects while correctly detecting/classifying known objects. We first point out that the scenario of OSOD considered in recent studies, which considers an unlimited variety of unknown objects similar to open-set recognition (OSR), has a fundamental issue. That is, we cannot determine what to detect and what not for such unlimited unknown objects, which is necessary for detection tasks. This issue leads to difficulty with the evaluation of methods' performance on unknown object detection. We then introduce a novel scenario of OSOD, which deals with only unknown objects that share the super-category with known objects. It has many real-world applications, e.g., detecting an increasing number of fine-grained objects. This new setting is free from the above issue and evaluation difficulty. Moreover, it makes detecting unknown objects more realistic owing to the visual similarity between known and unknown objects. We show through experimental results that a simple method based on the uncertainty of class prediction from standard detectors outperforms the current state-of-the-art OSOD methods tested in the previous setting.

Introduction: The "Objectness" Dilemma

In standard object detection, models operate within a closed set of categories. Open-Set Object Detection (OSOD) attempts to bridge the gap to the real world by accounting for "unknowns." However, the dominant research paradigm (OSOD-II) treats unknowns as an infinite set.

This creates a logical contradiction: detection requires a bounding box, which implies a definition of what is an "object." If "unknown" is unbounded, should a detector label every tire, tree branch, or pebble? This ambiguity makes it impossible to establish a ground truth for training or to use reliable metrics. Current metrics like Wilderness Impact (WI) merely measure how often a model is distracted by unknowns, rather than its ability to find them.

Rethinking OSOD: The OSOD-III Proposal

To resolve these issues, we categorize OSOD into three scenarios:

  1. OSOD-I: Detecting known objects while ignoring unknowns.
  2. OSOD-II: Detecting both known and unknown objects from any category (ill-posed).
  3. OSOD-III: Detecting known and unknown objects within a specific super-class.
Taxonomy of OSOD scenarios. OSOD-III defines the target space within a super-class, making the detection of unknowns a well-defined task.

In OSOD-III, the super-class (e.g., "Vehicles") defines the scope of "objectness." The model is trained on specific categories (e.g., "Car," "Truck") but must also detect and flag novel categories within that super-class (e.g., "Ambulance") as "unknown." This bounded approach allows researchers to use standard metrics like Average Precision (AP) for unknown classes.

Experimental Results and Insights

We benchmarked several state-of-the-art methods—including ORE, VOS, and OpenDet—on datasets like Open Images V6, CUB200, and MTSD.

Evaluation of state-of-the-art detection methods using Wilderness Impact (WI) and other performance metrics across varying confidence thresholds.

Key findings include:

  • Performance Gap: While models perform well on known classes, AP for unknown objects remains low (5-25%), highlighting the difficulty of distinguishing visually similar categories within a super-class.
  • Baseline Performance: A simple baseline using class prediction uncertainty often matches or exceeds the performance of complex OSOD-specific architectures.
  • Localization vs. Classification: Models are generally effective at localizing unknown objects with bounding boxes; the primary failure occurs in classifying them as "unknown" rather than misidentifying them as a known category.

Conclusion

By shifting the focus to super-category-level detection, OSOD-III transforms an ill-posed research problem into a practical engineering task. Future research should prioritize feature decorrelation to distinguish the general characteristics of a super-class from the specific features of known categories, enabling better identification of novel instances.

Summary updated

Abstract

Open-set object detection (OSOD) has recently attracted considerable attention. It is to detect unknown objects while correctly detecting/classifying known objects. We first point out that the scenario of OSOD considered in recent studies, which considers an unlimited variety of unknown objects similar to open-set recognition (OSR), has a fundamental issue. That is, we cannot determine what to detect and what not for such unlimited unknown objects, which is necessary for detection tasks. This issue leads to difficulty with the evaluation of methods' performance on unknown object detection. We then introduce a novel scenario of OSOD, which deals with only unknown objects that share the super-category with known objects. It has many real-world applications, e.g., detecting an increasing number of fine-grained objects. This new setting is free from the above issue and evaluation difficulty. Moreover, it makes detecting unknown objects more realistic owing to the visual similarity between known and unknown objects. We show through experimental results that a simple method based on the uncertainty of class prediction from standard detectors outperforms the current state-of-the-art OSOD methods tested in the previous setting.
BibTeX
@misc{hosoya2022more,
  author = {Yusuke Hosoya and Masanori Suganuma and Takayuki Okatani},
  title = {{More Practical Scenario of Open-set Object Detection: Open at Category Level and Closed at Super-category Level}},
  year = {2022},
  eprint = {2207.09775},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.09775},
}

Loading related content...

Figure