Sampling Bag of Views for Open-Vocabulary Object Detection

Hojun Choi, Junsuk Choe, Hyunjung ShimarXiv 2024 · DOI · arXiv

Abstract

Open-vocabulary object detection (OVD) aims to recognize objects beyond the training distribution by leveraging pre-trained Vision-Language Models (VLMs). While recent methods like BARON utilize a "bag-of-regions" to provide context, they often suffer from semantic noise due to spatial-only sampling and lack scale flexibility. We propose Sampling Bag of Views (SBV), which introduces an Adaptive Sampling Strategy (ASS) to group semantically related concepts and a hierarchical view structure with attention masking. SBV achieves a 2.6 AP gain on COCO novel categories and reduces CLIP-related computation by 80.3%, setting a new state-of-the-art in efficiency and accuracy.

Introduction

The primary challenge in open-vocabulary object detection is identifying categories not seen during training. Current approaches bridge this gap using VLMs like CLIP. A notable method, BARON, groups target objects with spatial neighbors to provide context. However, this approach is limited by Semantic Blindness—sampling irrelevant background noise—and Scale Rigidity—using fixed windows that fail to capture multi-scale relationships. These limitations result in "noisy compositional structures" that degrade performance and waste computational resources.

Methodology: Sampling Bag of Views (SBV)

SBV addresses these issues through three primary innovations designed to refine the contextual signal sent to the VLM.

Adaptive Sampling Strategy (ASS)

Instead of sampling based on pixel proximity, SBV treats the image as a graph-based canvas. Region Proposal Network (RPN) outputs serve as nodes, with edges representing semantic associations. This allows the model to explore the image and group contextually relevant "concepts" rather than arbitrary spatial neighbors.

Hierarchical Bag of Views

To handle varying object scales, SBV considers three hierarchical levels for each concept:

  1. Global View: The entire scene.
  2. Middle View: Merged related concepts.
  3. Local View: Individual object proposals.

A representation switching logic dynamically selects the most informative scale for each concept.

Attention Masking

To further purify the signal, SBV utilizes Noise Masks to suppress background features (e.g., sky, road) and View Masks to weight the importance of specific patches. This ensures the VLM focuses on meaningful object relationships rather than background clutter.

Comparison of the BARON baseline (top) and the proposed SBV architecture (bottom), highlighting the shift from spatial neighbor sampling to concept-based sampling and masking.

Experimental Results

The proposed method was evaluated on the OV-COCO and OV-LVIS benchmarks, demonstrating significant improvements in both detection quality and computational efficiency.

Accuracy and Efficiency

On the OV-COCO benchmark, SBV achieved 36.6 box AP50 on novel categories, a 2.6-point improvement over the BARON baseline. Crucially, the method reduced CLIP-related computation from 55.3 PetaFLOPs to 10.9 PetaFLOPs, representing an 80.3% efficiency gain.

Qualitative Analysis

Visual comparisons highlight SBV's ability to avoid the pitfalls of over-sampling and missing concepts. By focusing on semantically linked regions, SBV produces cleaner, more discriminative feature maps for novel objects compared to the scattered responses of previous models.

Qualitative comparison of BARON vs. SBV across different sampling challenges: (a) Over-sampled, (b) Missing concepts, and (c) Over-represented.

Conclusion

SBV demonstrates that semantic relevance is a superior metric for contextual sampling compared to spatial proximity. By integrating adaptive sampling with hierarchical views and masking, the model achieves state-of-the-art performance while drastically reducing the computational overhead associated with VLMs. Future work will explore tuning the global context weight for dense scenes and extending the "bag of views" concept to video-based detection.

Summary updated

Abstract

Existing open-vocabulary object detection (OVD) develops methods for testing unseen categories by aligning object region embeddings with corresponding VLM features. A recent study leverages the idea that VLMs implicitly learn compositional structures of semantic concepts within the image. Instead of using an individual region embedding, it utilizes a bag of region embeddings as a new representation to incorporate compositional structures into the OVD task. However, this approach often fails to capture the contextual concepts of each region, leading to noisy compositional structures. This results in only marginal performance improvements and reduced efficiency. To address this, we propose a novel concept-based alignment method that samples a more powerful and efficient compositional structure. Our approach groups contextually related ``concepts'' into a bag and adjusts the scale of concepts within the bag for more effective embedding alignment. Combined with Faster R-CNN, our method achieves improvements of 2.6 box AP50 and 0.5 mask AP over prior work on novel categories in the open-vocabulary COCO and LVIS benchmarks. Furthermore, our method reduces CLIP computation in FLOPs by 80.3% compared to previous research, significantly enhancing efficiency. Experimental results demonstrate that the proposed method outperforms previous state-of-the-art models on the OVD datasets.
BibTeX
@misc{choi2024sampling,
  author = {Hojun Choi and Junsuk Choe and Hyunjung Shim},
  title = {{Sampling Bag of Views for Open-Vocabulary Object Detection}},
  year = {2024},
  eprint = {2412.18273},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.18273},
}
Figure