Fine-Grained Open-Vocabulary Object Detection with Fined-Grained Prompts: Task, Dataset and Benchmark

Ying Liu, Yijing Hua, Haojiang Chai, Yanbo Wang, TengQi Ye

Abstract

Open-vocabulary detectors are proposed to locate and recognize objects in novel classes. However, variations in vision-aware language vocabulary data used for open-vocabulary learning can lead to unfair and unreliable evaluations. Recent evaluation methods have attempted to address this issue by incorporating object properties or adding locations and characteristics to the captions. Nevertheless, since these properties and locations depend on the specific details of the images instead of classes, detectors can not make accurate predictions without precise descriptions provided through human annotation. This paper introduces 3F-OVD, a novel task that extends supervised fine-grained object detection to the open-vocabulary setting. Our task is intuitive and challenging, requiring a deep understanding of fine-grained captions and careful attention to fine-grained details in images in order to accurately detect fine-grained objects. Additionally, due to the scarcity of qualified fine-grained object detection datasets, we have created a new dataset, NEU-171K, tailored for both supervised and open-vocabulary settings. We benchmark state-of-the-art object detectors on our dataset for both settings and propose a simple yet effective post-processing technique to improve performance.

What Problem Does This Paper Address?

Standard Open-Vocabulary Detection (OVD) aims to find objects the model wasn't explicitly trained on by using text descriptions. However, current evaluation methods are often inconsistent. Some rely on image-specific captions (visual grounding), which requires expensive human annotation for every single image. Others suffer from data leakage, where "novel" classes were actually seen by the model during its massive web-scale pre-training.

Furthermore, detecting fine-grained differences—like distinguishing between a "Ford Focus" and a "Nissan Tiida" rather than just a generic "car"—is incredibly difficult. Existing models often get confused by long, descriptive captions, frequently triggering on individual words (like "logo" or "text") rather than the whole object, leading to a sea of false-positive bounding boxes.

Comparison of different object detection tasks and the proposed 3F-OVD approach.

Methods

The researchers propose three main contributions to advance the field:

  1. The 3F-OVD Task: Unlike previous tasks that use unique captions for every image, 3F-OVD uses a single, detailed descriptive caption for each class. This mimics a realistic scenario where a user provides a description of a specific item they want the system to find across many images.
  2. NEU-171K Dataset: A large-scale dataset specifically designed for fine-grained detection across two domains: Vehicles (captured via handheld cameras and dashcams) and Retail Products (captured in controlled lab settings).
  3. A Novel Post-processing Trick: They observed that OVD models often predict bounding boxes for individual words within a caption (e.g., a tiny box for the "text" on a cereal box). Their method filters these out by calculating the proportion of overlapping areas between boxes and eliminating boxes that are "excessively large or small" based on domain-specific constraints.

Datasets

The paper introduces and utilizes:

  • NEU-171K (New): 145,825 images and 676,471 bounding boxes across 719 fine-grained classes. It is split into NEU-171K-C (Cars) and NEU-171K-RP (Retail Products).
  • Benchmark Comparisons: The authors compare their dataset against existing ones like FGVD (Vehicles), RPC (Retail), CUB-200 (Birds), and LVIS/COCO (General objects).

Key Insights and Ideas

The "Word-Level" Trap

Current state-of-the-art OVD models (like ViLD or GroundingDINO) don't actually "understand" a full descriptive sentence as a single concept. Instead, they often act as word-detectors, placing boxes around anything that matches a specific noun in the prompt. This leads to fragmented detections where parts of an object are identified as the whole.

Example of the "Word-Level Trap" in retail product detection, where models predict fragmented boxes for individual words.

Fine-Grained is the True Test

Open-vocabulary models perform much worse on fine-grained tasks than on general object detection. This suggests that while models are good at identifying broad categories, they struggle with the nuances required for professional or industrial applications.

Class-Level Captions Reduce Bias

By using fixed captions per class, the benchmark ensures that the model's performance is based on its ability to generalize visual features from text, rather than relying on image-specific hints provided by a human annotator.

Key Results

The authors benchmarked several leading models, including ViLD, Detic, and GroundingDINO:

  • Task Difficulty: Models performed significantly worse on the 3F-OVD protocol compared to older protocols (FG-OVD and OV-VG), proving that 3F-OVD is a much more challenging and rigorous test of a model's understanding.
  • Post-processing Gains: Their simple filtering trick provided massive boosts. For example, ViLD’s performance (mAP) on retail products jumped by 41.3% simply by filtering out the "fragmented" detections caused by long captions.
  • Domain Differences: Models generally performed better on the Retail (RP) dataset than the Car (C) dataset, likely because the retail images have static backgrounds and clearer visual distinctions.

Limitations and Future Work

The authors note that current OVD detectors still struggle to predict at the "caption level." Even with their post-processing trick, the underlying models are still fundamentally biased toward token-level matching.

Future work is needed to develop architectures that can truly integrate a full descriptive paragraph into a single visual concept without getting distracted by individual words. Additionally, while NEU-171K is large, expanding it to more diverse domains (like medical imaging or specialized machinery) would further test the limits of fine-grained OVD.

Summary updated

Abstract

Open-vocabulary detectors are proposed to locate and recognize objects in novel classes. However, variations in vision-aware language vocabulary data used for open-vocabulary learning can lead to unfair and unreliable evaluations. Recent evaluation methods have attempted to address this issue by incorporating object properties or adding locations and characteristics to the captions. Nevertheless, since these properties and locations depend on the specific details of the images instead of classes, detectors can not make accurate predictions without precise descriptions provided through human annotation. This paper introduces 3F-OVD, a novel task that extends supervised fine-grained object detection to the open-vocabulary setting. Our task is intuitive and challenging, requiring a deep understanding of Fine-grained captions and careful attention to Fine-grained details in images in order to accurately detect Fine-grained objects. Additionally, due to the scarcity of qualified fine-grained object detection datasets, we have created a new dataset, NEU-171K, tailored for both supervised and open-vocabulary settings. We benchmark state-of-the-art object detectors on our dataset for both settings. Furthermore, we propose a simple yet effective post-processing technique.
BibTeX
@misc{liu2025fine,
  author = {Ying Liu and Yijing Hua and Haojiang Chai and Yanbo Wang and TengQi Ye},
  title = {{Fine-Grained Open-Vocabulary Object Detection with Fined-Grained Prompts: Task, Dataset and Benchmark}},
  year = {2025},
  eprint = {2503.14862},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.14862},
  howpublished = {arXiv},
}
Figure