Abstract
Open-vocabulary object detection (OVD) aims to recognize objects beyond the training distribution by leveraging pre-trained Vision-Language Models (VLMs). While recent methods like BARON utilize a "bag-of-regions" to provide context, they often suffer from semantic noise due to spatial-only sampling and lack scale flexibility. We propose Sampling Bag of Views (SBV), which introduces an Adaptive Sampling Strategy (ASS) to group semantically related concepts and a hierarchical view structure with attention masking. SBV achieves a 2.6 AP gain on COCO novel categories and reduces CLIP-related computation by 80.3%, setting a new state-of-the-art in efficiency and accuracy.
Introduction
The primary challenge in open-vocabulary object detection is identifying categories not seen during training. Current approaches bridge this gap using VLMs like CLIP. A notable method, BARON, groups target objects with spatial neighbors to provide context. However, this approach is limited by Semantic Blindness—sampling irrelevant background noise—and Scale Rigidity—using fixed windows that fail to capture multi-scale relationships. These limitations result in "noisy compositional structures" that degrade performance and waste computational resources.
Methodology: Sampling Bag of Views (SBV)
SBV addresses these issues through three primary innovations designed to refine the contextual signal sent to the VLM.
Adaptive Sampling Strategy (ASS)
Instead of sampling based on pixel proximity, SBV treats the image as a graph-based canvas. Region Proposal Network (RPN) outputs serve as nodes, with edges representing semantic associations. This allows the model to explore the image and group contextually relevant "concepts" rather than arbitrary spatial neighbors.
Hierarchical Bag of Views
To handle varying object scales, SBV considers three hierarchical levels for each concept:
- Global View: The entire scene.
- Middle View: Merged related concepts.
- Local View: Individual object proposals.
A representation switching logic dynamically selects the most informative scale for each concept.
Attention Masking
To further purify the signal, SBV utilizes Noise Masks to suppress background features (e.g., sky, road) and View Masks to weight the importance of specific patches. This ensures the VLM focuses on meaningful object relationships rather than background clutter.
Experimental Results
The proposed method was evaluated on the OV-COCO and OV-LVIS benchmarks, demonstrating significant improvements in both detection quality and computational efficiency.
Accuracy and Efficiency
On the OV-COCO benchmark, SBV achieved 36.6 box AP50 on novel categories, a 2.6-point improvement over the BARON baseline. Crucially, the method reduced CLIP-related computation from 55.3 PetaFLOPs to 10.9 PetaFLOPs, representing an 80.3% efficiency gain.
Qualitative Analysis
Visual comparisons highlight SBV's ability to avoid the pitfalls of over-sampling and missing concepts. By focusing on semantically linked regions, SBV produces cleaner, more discriminative feature maps for novel objects compared to the scattered responses of previous models.
Conclusion
SBV demonstrates that semantic relevance is a superior metric for contextual sampling compared to spatial proximity. By integrating adaptive sampling with hierarchical views and masking, the model achieves state-of-the-art performance while drastically reducing the computational overhead associated with VLMs. Future work will explore tuning the global context weight for dense scenes and extending the "bag of views" concept to video-based detection.
Summary updated