Abstract
Open-vocabulary object detection (OVOD) aims to identify and localize objects beyond a predefined set of training categories by leveraging vision-language models (VLMs). While current methods excel at classifying novel objects, they often struggle with precise localization and segmentation due to their reliance on class-agnostic detection heads. This paper introduces CondHead, a novel dynamic network design that makes bounding box and mask prediction "class-aware." By conditionally parameterizing detection heads based on semantic embeddings, CondHead enables the generalization of geometric knowledge from base to novel categories, significantly improving localization performance in open-vocabulary scenarios.
Introduction
Traditional object detectors operate under a closed-set assumption, limited to the categories present in their training data. OVOD breaks this barrier by using VLMs like CLIP to recognize novel objects based on textual descriptions. However, a significant performance gap persists in the localization and segmentation of these unseen classes.
Most existing OVOD frameworks utilize class-agnostic heads, which treat all objects as geometrically identical. This approach fails to exploit the inherent structural relationships between categories. For instance, if a model learns to segment the spindly legs of a chicken, that geometric prior should theoretically assist in segmenting the similar legs of an ibis, even if the latter was never seen during training. CondHead bridges this gap by moving from class-agnostic to class-conditional prediction.
Methodology: The CondHead Architecture
CondHead replaces standard fixed detection heads with a dual-stream dynamic architecture. The parameters of these heads are generated on-the-fly based on the semantic embedding of a category name, allowing the model to adapt its geometric predictions to the specific structure of the object.
Dual-Stream Dynamic Heads
The architecture consists of two complementary streams:
- Dynamically Aggregated Head: This stream maintains a set of "static expert" heads. A controller network analyzes the semantic embedding to generate weights for aggregating these experts. This allows the model to learn specialized prediction patterns for diverse object geometries.
- Dynamically Generated Head: This stream directly predicts the weights of a lightweight network from the semantic embedding. It is designed to capture broader class-specific traits, such as typical aspect ratios or color distributions.
To encourage specialization among the aggregated experts, the authors employ Temperature Annealing during training. By gradually lowering the softmax temperature, the model is forced to select specific experts for specific semantic regions, ensuring a diverse set of geometric priors.
Key Insights
The central premise of CondHead is that semantics encode geometry. Semantic embeddings from models like CLIP contain latent structural information; categories like "chair" and "sofa" are proximate in embedding space not just due to their functional similarity, but because they share physical affordances like flat seats and backs. CondHead exploits this latent information to transfer geometric "know-how" across the vocabulary.
Furthermore, the dual-stream approach balances complexity and efficiency. While the aggregated head handles complex geometric patterns, the generated head provides general stability. Despite this dynamic generation, CondHead adds only approximately 1.1% computational overhead compared to standard class-agnostic baselines.
Experimental Results
CondHead was evaluated on several benchmarks, including LVIS, COCO, PASCAL VOC, and Objects365. The results demonstrate consistent improvements across various base architectures:
- Novel Category Gains: CondHead improved the RegionClip baseline by 3.0 AP (Average Precision) on novel categories in the LVIS dataset.
- Architecture Agnosticism: Across OVR-CNN, ViLD, and RegionClip, the method provided an average improvement of 2.8 AP for both bounding box regression and mask segmentation tasks.
- Cross-Dataset Robustness: When trained on LVIS and tested on PASCAL VOC and Objects365, the model showed gains of +2.4 AP and +1.4 AP respectively, proving that the learned geometric priors are highly generalizable.
Conclusion
CondHead demonstrates that making detection heads class-aware via semantic conditioning is a powerful strategy for open-vocabulary tasks. By bridging the geometric gap between base and novel categories, it allows detectors to localize objects they have never seen with unprecedented accuracy. Future research may focus on tighter integration of visual features into the parameter generation process to further close the perception gap in complex, occluded environments.
Summary updated