Open-vocabulary vs. Closed-set: Best Practice for Few-shot Object Detection Considering Text Describability

Yusuke Hosoya, Masanori Suganuma, Takayuki Okatani

Introduction

Few-shot object detection (FSOD) aims to detect novel objects using only a handful of examples. Recently, Open-Vocabulary Detection (OVD) models, which leverage massive vision-language models like CLIP, have been proposed as a solution. However, a practical gap exists: many real-world targets, such as industrial defects or medical lesions, are difficult to describe accurately with words. This paper investigates whether the complexity of OVD is justified for these "low-describability" classes compared to traditional Closed-set Object Detection (COD).

Methodology: The Describability Metric

The authors introduce a novel way to categorize detection tasks based on "text describability." They use zero-shot image classification accuracy with CLIP as a proxy indicator. If CLIP can easily identify an object's category from a crop based on its name, the class is considered "highly describable."

Using this metric, 35 diverse datasets from the Object Detection in the Wild (ODinW) collection were categorized into three tiers:

  • S1 (High describability): Common objects like "cat" or "dog."
  • S2 (Medium describability): Objects with moderate naming clarity.
  • S3 (Low describability): Specialized targets like "blood cells," "sign language," or "industrial parts."
Zero-shot accuracy as a proxy for text describability across 35 datasets, categorized into S1, S2, and S3 tiers.

Results: OVD vs. COD Performance

The study compared state-of-the-art OVD models (e.g., GLIP, F-ViT) against COD models (e.g., DyHead, Faster R-CNN). The results highlight a clear "describability threshold": OVD's primary advantage is its linguistic prior, and when this prior is weak, OVD effectively reverts to a standard detector.

AP ratio of OVD over COD across different shots (K) and describability tiers. The advantage of OVD diminishes as describability decreases.

Key performance findings include:

  • S1 (Easy to Describe): OVD significantly outperforms COD. At 1-shot (K=1), GLIP(A) achieved 37.4 AP compared to DyHead’s 29.0 AP.
  • S3 (Hard to Describe): The gap vanishes. At 3-shots (K=3), DyHead (COD) achieved 39.7 AP, while GLIP(A) (OVD) achieved 39.2 AP.

Discussion: The Impact of Data Scaling

A surprising finding is that while OVD models benefit from diverse web-scale data for common objects, this can be counterproductive for hard-to-describe classes. For instance, training GLIP on 5.46M images (including web captions) improved S1 performance but caused S3 performance to drop from 39.2 AP to 34.9 AP. The "noise" of general web data appears to dilute the model's ability to learn the specific visual features required for niche S3 tasks. Additionally, for FSOD with a significant domain gap, simply finetuning all parameters ("Full-FT") consistently outperformed specialized few-shot algorithms like TFA or FSCE.

Conclusion and Practical Guidance

For practitioners, the choice between OVD and COD should depend on the target domain. If objects are standard and easily described (e.g., cars, people), OVD is the superior choice. However, for specialized domains like medical imaging or industrial inspection, traditional COD methods are likely more efficient and just as accurate. Future work may explore "visual prompting" to bridge the gap for classes that are hard to describe but easy to show visually.

Summary updated

Abstract

Open-vocabulary object detection (OVD), detecting specific classes of objects using only their linguistic descriptions (e.g., class names) without any image samples, has garnered significant attention. However, in real-world applications, the target class concepts is often hard to describe in text and the only way to specify target objects is to provide their image examples, yet it is often challenging to obtain a good number of samples. Thus, there is a high demand from practitioners for few-shot object detection (FSOD). A natural question arises: Can the benefits of OVD extend to FSOD for object classes that are difficult to describe in text? Compared to traditional methods that learn only predefined classes (referred to in this paper as closed-set object detection, COD), can the extra cost of OVD be justified? To answer these questions, we propose a method to quantify the ``text-describability'' of object detection datasets using the zero-shot image classification accuracy with CLIP. This allows us to categorize various OD datasets with different text-describability and emprically evaluate the FSOD performance of OVD and COD methods within each category. Our findings reveal that: i) there is little difference between OVD and COD for object classes with low text-describability under equal conditions in OD pretraining; and ii) although OVD can learn from more diverse data than OD-specific data, thereby increasing the volume of training data, it can be counterproductive for classes with low-text-describability. These findings provide practitioners with valuable guidance amidst the recent advancements of OVD methods.
BibTeX
@misc{hosoya2024open,
  author = {Yusuke Hosoya and Masanori Suganuma and Takayuki Okatani},
  title = {{Open-vocabulary vs. Closed-set: Best Practice for Few-shot Object Detection Considering Text Describability}},
  year = {2024},
  eprint = {2410.15315},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.15315},
}
Figure