Abstract
We propose a new setting for detecting unseen objects called Zero-shot Annotation object Detection (ZAD). It expands the zero-shot object detection setting by allowing the novel objects to exist in the training images and restricts the additional information the detector uses to novel category names. To solve problems in existing distillation-based methods, we propose Efficient feature distillation for Zero-shot Annotation object Detection (EZAD). EZAD adapts CLIP’s feature space by re-normalizing CLIP, uses CLIP to generate distillation proposals with potential novel category names, and takes advantage of semantic meaning for regression. EZAD outperforms previous methods on COCO by 4% with a much shorter training schedule and achieves a 3% improvement on the LVIS dataset.
Introduction: The ZAD Setting
Detecting objects that were not present during training (zero-shot detection) is a critical challenge in computer vision. Traditionally, this is approached via two settings:
- Zero-Shot Detection (ZSD): A strict setting where novel objects cannot appear in training images at all.
- Open-Vocabulary Detection (OVD): A more flexible setting that often relies on massive external image-caption datasets.
The authors introduce Zero-shot Annotation object Detection (ZAD) as a more realistic middle ground. In ZAD, novel objects can appear in training images but are not annotated. The detector is provided with the names of these novel categories to guide the learning process.
Existing distillation methods, which transfer knowledge from large-scale vision-language models like CLIP into a detector, face two hurdles: a domain gap between CLIP's training data and detection datasets, and a bias in Region Proposal Networks (RPN) toward base categories, which prevents the model from "seeing" novel objects during training.
The EZAD Framework
To address these issues, the authors propose EZAD (Efficient Zero-shot Annotation Detection). The framework consists of four main stages designed to bridge the domain gap and improve novel object discovery.
1. Feature Space Adaptation
Instead of expensive fine-tuning of the entire CLIP model, EZAD adapts CLIP’s feature space by fine-tuning only the LayerNorm layers using the base categories of the target detection dataset. This "re-normalization" aligns CLIP features with the detection domain, significantly improving classification accuracy without high computational costs.
2. CLIP Proposal Selection
Standard RPNs are trained only on base categories and often fail to propose regions containing novel objects. EZAD overcomes this by using CLIP offline to generate distillation proposals. Because CLIP is open-vocabulary, it can identify potential novel object regions that the RPN would miss.
3. CLIP Proposal Feature Distillation
The model distills knowledge from the adapted CLIP-Visual features into the detection backbone. By using the CLIP-generated proposals, the distillation process is no longer biased solely toward base categories, allowing the model to learn features relevant to novel objects.
4. Semantic-based Regression
EZAD incorporates the text embedding of the predicted category into the bounding box regressor. By combining visual features with semantic knowledge (e.g., the typical shape of a "dog" versus a "car"), the model can learn category-specific geometric priors, leading to more accurate localization.
Experimental Results
EZAD was evaluated on MS-COCO and LVIS datasets, demonstrating both improved accuracy and training efficiency.
- MS-COCO: EZAD achieved 31.6 AP50 on novel categories, a 4% improvement over the previous state-of-the-art method, ViLD (27.6 AP50).
- Efficiency: Remarkably, EZAD reached these results in only 36 epochs, compared to the 107 epochs required by ViLD—a 3x speedup in training time.
- LVIS: The method showed a consistent ~3% improvement over baselines. The authors also introduced a new "LVIS-Fbase" split to provide a more robust benchmark for zero-shot performance compared to the sparse "Rare" split.
Conclusion
EZAD demonstrates that efficient adaptation of vision-language models can significantly enhance zero-shot object detection. By addressing the domain gap through LayerNorm tuning and the RPN bias through CLIP-guided proposals, EZAD achieves state-of-the-art performance with a fraction of the training time required by previous distillation-based approaches.
Summary updated