Introduction
Traditional object detection models are typically constrained by a "closed-set" paradigm, where they can only recognize categories present in their training data. However, real-world visual data follows a long-tail distribution, containing a vast number of rare or "novel" objects for which labeled data is scarce. While Open Vocabulary Detection (OVD) aims to address this, many existing solutions require significant computational resources for retraining. This paper proposes a training-free, cooperative framework that leverages off-the-shelf foundational models—such as CLIP and SAM—to transform standard detectors into robust open-set systems capable of Novel Object Detection (NOD).
Methodology: A Cooperative Framework
The proposed system orchestrates multiple specialized models in a modular pipeline to identify and localize both known and novel objects without requiring expensive retraining.
The pipeline consists of four primary stages:
- Initialization: The process begins with two detectors. A standard Mask-RCNN identifies known objects and flags "background" regions that may contain novel items. Simultaneously, Grounding DINO (GDINO), an open-set detector, provides initial candidates for potential objects.
- Unknown Object Labeling: To classify the background regions identified by Mask-RCNN, the system utilizes the CLIP vision-language model. Accuracy is enhanced through a Synonym Averaged Embedding Generator (SAEG), which averages text embeddings of various synonyms to create a more robust class prototype.
- Refinement with SAM: To improve localization, bounding boxes from the initial detectors are used as prompts for the Segment Anything Model (SAM). SAM generates precise masks, which are then converted back into tightly fitted bounding boxes.
- Score Refinement Module (SRM): The SRM filters detections by combining confidence scores from the detector with a "mask quality" score from SAM. This ensures that only well-localized and high-confidence detections are retained.
Experimental Evaluation
The framework was evaluated on several challenging benchmarks:
- LVIS v1: A dataset with 1,203 categories designed for long-tail recognition. The authors used a split of 80 known and 1,123 novel classes.
- COCO OVD Split: A standard benchmark (48 known, 17 novel classes) for comparing against other OVD methods.
- lvis_val_subset: A curated subset of 745 images used for ablation studies to ensure all classes were represented during testing.
Key Results and Insights
The cooperative approach demonstrates significant improvements in detecting novel objects. By treating the "background" of closed-set detectors as a source of potential novel items, the system effectively "discovers" objects that traditional models ignore.
As shown in the performance comparison, the addition of the proposed cooperative module significantly boosts the Novel AP for both Mask-RCNN and GDINO. Specifically, on the LVIS benchmark, the method achieved 17.42 mAP for novel objects, a substantial leap from the previous state-of-the-art (5.42 mAP). On the COCO OVD split, the system reached 49.6 Novel AP50, outperforming many methods that require intensive training. Furthermore, the framework achieved a 3.5% absolute gain in recall over previous best methods, indicating superior localization and naming capabilities.
Conclusion and Future Work
The modular nature of this framework allows for "plug-and-play" updates as better foundational models emerge. However, the sequential execution of multiple large models (GDINO, CLIP, SAM, Mask-RCNN) results in high computational overhead and slower inference speeds compared to unified models. Future research will focus on distilling the knowledge from this complex pipeline into smaller, more efficient models suitable for real-time applications. Additionally, addressing potential data leakage inherent in large-scale pre-trained models remains a priority for the field.
Summary updated