Abstract: We propose a new setting for detecting unseen objects called Zero-shot Annotation object Detection (ZAD). It expands the zero-shot object detection setting by allowing the novel objects to exist in the training images and restricts the additional information the detector uses to novel category names. Recently, to detect unseen objects, large-scale vision-language models (e.g., CLIP) are leveraged by different methods. The distillation-based methods have good overall performance but suffer from a long training schedule caused by two factors. First, existing work creates distillation regions biased to the base categories, which limits the distillation of novel category information. Second, directly using the raw feature from CLIP for distillation neglects the domain gap between the training data of CLIP and the detection datasets, which makes it difficult to learn the mapping from the image region to the vision-language feature space. To solve these problems, we propose Efficient feature distillation for Zero-shot Annotation object Detection (EZAD). Firstly, EZAD adapts the CLIP's feature space to the target detection domain by re-normalizing CLIP; Secondly, EZAD uses CLIP to generate distillation proposals with potential novel category names to avoid the distillation being overly biased toward the base categories. Finally, EZAD takes advantage of semantic meaning for regression to further improve the model performance. As a result, EZAD outperforms the previous distillation-based methods in COCO by 4% with a much shorter training schedule and achieves a 3% improvement on the LVIS dataset.
Introduction
Traditional object detection models operate under a "closed-world" assumption, where they can only identify categories explicitly labeled during training. To address the need for more flexible systems, researchers have developed Zero-shot Detection (ZSD) and Open-Vocabulary Detection (OVD). However, ZSD is often too restrictive, assuming novel objects never appear in training images, while OVD frequently relies on massive external datasets or image-caption pairs that may not be readily available for specific domains.
This paper introduces a more realistic middle ground: Zero-shot Annotation object Detection (ZAD). In the ZAD setting, novel objects are permitted to appear in training images (though they remain unlabeled), and the model is provided only with the names of these new categories.
The EZAD Framework
Existing distillation methods, which attempt to "teach" a detector to mimic vision-language models like CLIP, face two primary hurdles: the domain gap between web-scraped CLIP data and specific detection datasets, and a bias toward base categories caused by standard Region Proposal Networks (RPN). We propose EZAD (Efficient Zero-shot Annotation object Detection) to overcome these challenges through three core innovations.
1. CLIP Feature Adaptation
Instead of treating CLIP as a frozen black box, EZAD fine-tunes only its LayerNorm layers using base category instances from the target dataset. This "re-normalization" bridges the domain gap, adapting CLIP to the specific visual characteristics of the detection data (e.g., small or occluded objects) without sacrificing its broad general knowledge.
2. CLIP Proposals for Distillation
Standard RPNs are trained only on labeled base categories and often fail to propose regions containing novel objects. To break this cycle, EZAD uses CLIP offline to scan images and identify regions likely to contain any object from a dictionary of both base and novel names. These "CLIP Proposals" ensure the model distills knowledge from regions containing novel objects, preventing the training process from becoming overly biased toward known categories.
3. Semantic-based Regressor
While standard detectors use a generic head for bounding box refinement, EZAD incorporates the text embedding of the predicted category into the regression head. This allows the model to leverage semantic knowledge to more accurately localize and "shrink-wrap" boxes around previously unseen objects.
Experimental Results
We evaluated EZAD on MS-COCO and LVIS datasets. On MS-COCO (split into 48 base and 17 novel categories), EZAD achieved a 31.6 AP50 on novel categories—a 4% improvement over the previous state-of-the-art, ViLD. Notably, EZAD reached this performance while training 3x faster (36 epochs vs. 100+ for competitors).
On the LVIS dataset, specifically the LVIS-Fbase split designed to better represent real-world performance, EZAD outperformed ViLD by approximately 3% across both base and novel categories. Qualitative analysis confirms that EZAD is significantly more accurate at correctly labeling novel objects, such as "sinks" or "benches," compared to models that lack feature space adaptation.
Conclusion
EZAD demonstrates that efficient zero-shot detection is achievable by focusing on normalization and unbiased proposal generation. By adapting CLIP's feature space through LayerNorm and utilizing CLIP-driven proposals, the model escapes the limitations of base-category-biased RPNs. Future work will explore reducing the noise in CLIP proposals and extending these distillation efficiencies to one-stage detectors and Transformer-based architectures.
Summary updated