Open-vocabulary vs. Closed-set: Best Practice for Few-shot Object Detection Considering Text Describability

Yusuke Hosoya, Masanori Suganuma, Takayuki OkataniSubmission to NeurIPS 2024 Track Datasets and Benchmarks · Rejected

TL;DR

While Open-Vocabulary Object Detection (OVD) shines when classes have clear textual names, its performance advantages largely vanish when target objects are difficult to describe in text (such as industrial defects or specialized symbols). Under controlled few-shot settings, traditional Closed-Set Object Detection (COD) performs just as well as—and sometimes better than—OVD on low-describability concepts, showing that the extra computational complexity and web-scale pre-training of OVD are not always justified.

What Problem Does This Paper Address?

In real-world computer vision, practitioners often need to detect novel object classes using only a handful of examples (Few-Shot Object Detection, or FSOD). Recently, Open-Vocabulary Object Detection (OVD) models have gained massive popularity because they leverage vision-language pre-training (e.g., CLIP, BERT) to detect unseen objects from text prompts alone.

However, many critical real-world targets—such as subtle manufacturing defects, specialized medical lesions, or abstract symbols—are notoriously difficult to specify using natural language. For these concepts, users must provide image examples rather than descriptions. This raises key practical questions:

  1. Do the benefits of OVD transfer to few-shot detection when classes have low "text-describability"?
  2. Does the added computational overhead of OVD models justify their use over traditional Closed-Set Object Detection (COD) methods?

Methods

To systematically evaluate how "verbalizability" affects detection, the authors introduce a proxy benchmark pipeline:

  1. Quantifying "Text-Describability": Ground-truth bounding box crops of objects are passed through CLIP. For text prompts, the class names across all evaluated datasets are pooled into a single global candidate vocabulary $C$. Zero-shot image classification accuracy over this shared candidate pool serves as the metric for how readily CLIP associates the visual concepts with their text labels.
  2. Controlled Model Comparison: The authors match detector backbones and architectures across both paradigms:
  3. One-stage detectors: DyHead (COD) vs. GLIP(A) (OVD, which adapts DyHead with a BERT text encoder).
  4. Two-stage detectors: Faster R-CNN (COD) vs. F-ViT (OVD, built on Faster R-CNN using frozen EVA-CLIP representations).
  5. FSOD Finetuning Protocols: They compare standard full fine-tuning (Full-FT) against popular few-shot adaptation methods: TFA (freezing the backbone/proposal networks and tuning only the heads) and FSCE (contrastive proposal encoding).

Datasets

  • Evaluation Benchmark: 35 datasets from ODinW (Object Detection in the Wild, drawn from Roboflow), partitioned into three 11-to-12-dataset splits based on CLIP zero-shot classification performance:
  • S1 (High Describability): Common objects (e.g., PASCAL VOC, standard vehicles).
  • S2 (Medium Describability): Specialized natural categories (e.g., underwater sea life, fine-grained plant diseases).
  • S3 (Low Describability): Abstract or niche symbols (e.g., blood cell types, sign language alphabet, font recognition).
  • Pre-training Datasets: Objects365-V1 (O365) and COCO for base detectors. For scaled OVD comparisons, GLIP incorporates GoldG (phrase grounding) and image-caption datasets (Conceptual Captions and SBU).

Key Insights and Ideas

  • OVD’s edge is tied to text describability: For everyday categories (S1), OVD leverages linguistic priors to dominate COD. But as text describability plummets (S3), text representations cease to provide meaningful guidance, collapsing OVD’s performance down to standard COD baselines.
  • Web-scale data scaling can backfire: Expanding pre-training from detection boxes to millions of noisy image-text pairs (e.g., scaling GLIP(A) to GLIP using 5.46M images) boosts accuracy on standard classes (S1/S2) but actively degrades performance on low-describability classes (S3).
  • Full fine-tuning beats parameter-efficient FSOD heuristics: Popular few-shot methods like TFA and FSCE (which freeze feature extractors to prevent overfitting) perform significantly worse than updating all model parameters (Full-FT) when dealing with large domain shifts or low-describability concepts.

Key Results

Under a 3-shot ($K=3$) full fine-tuning evaluation:

Split DyHead (COD) GLIP(A) (OVD) Faster R-CNN (COD) F-ViT (OVD)
S1 (High) 39.2 AP 44.6 AP 36.2 AP 45.5 AP
S2 (Medium) 33.9 AP 37.1 AP 31.4 AP 32.9 AP
S3 (Low) 39.7 AP 39.2 AP 38.1 AP 32.0 AP
  • Diminishing Returns: On S3, GLIP(A) performs on par with DyHead (39.2 vs. 39.7 AP), while Faster R-CNN beats its OVD counterpart F-ViT by 6.1 AP.
  • Pre-training Scale Inversion: On S3 at $K=3$, GLIP pre-trained on 5.46M image-caption/grounding pairs drops to 34.9 AP, falling substantially behind the smaller O365-only GLIP(A) (39.2 AP) and COD DyHead (39.7 AP).
  • Finetuning Impact: On S3, TFA causes DyHead to collapse from 39.7 AP to 16.0 AP, and GLIP(A) from 39.2 AP to 10.7 AP, demonstrating that high-layer-only tuning fails under cross-domain few-shot shifts.

Limitations and Future Work

  • Proxy Confounding: The zero-shot CLIP accuracy metric cannot strictly separate whether a category is inherently hard to describe in words or simply rare/absent from web-scraped multi-modal training sets.
  • Prompt Engineering: The metric uses basic templates ("an image of {class name}"); more expressive multi-modal prompt engineering or detailed textual descriptions were not explored.
  • Future Directions: The authors leave deeper analytical breakdowns of verbalizability vs. visual rarity to future work, alongside investigating visual prompt learning methods designed to bridge the gap for low-describability domains.

Summary updated

Abstract

Open-vocabulary object detection (OVD), detecting specific classes of objects using only their linguistic descriptions (e.g., class names) without any image samples, has garnered significant attention. However, in real-world applications, the target class concepts is often hard to describe in text and the only way to specify target objects is to provide their image examples, yet it is often challenging to obtain a good number of samples. Thus, there is a high demand from practitioners for few-shot object detection (FSOD). A natural question arises: Can the benefits of OVD extend to FSOD for object classes that are difficult to describe in text? Compared to traditional methods that learn only predefined classes (referred to in this paper as closed-set object detection, COD), can the extra cost of OVD be justified? To answer these questions, we propose a method to quantify the ``text-describability'' of object detection datasets using the zero-shot image classification accuracy with CLIP. This allows us to categorize various OD datasets with different text-describability and emprically evaluate the FSOD performance of OVD and COD methods within each category. Our findings reveal that: i) there is little difference between OVD and COD for object classes with low text-describability under equal conditions in OD pretraining; and ii) although OVD can learn from more diverse data than OD-specific data, thereby increasing the volume of training data, it can be counterproductive for classes with low-text-describability. These findings provide practitioners with valuable guidance amidst the recent advancements of OVD methods.
BibTeX
@misc{hosoya2024open,
  author = {Yusuke Hosoya and Masanori Suganuma and Takayuki Okatani},
  title = {{Open-vocabulary vs. Closed-set: Best Practice for Few-shot Object Detection Considering Text Describability}},
  year = {2024},
  eprint = {2410.15315},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.15315},
}
Figure