Zero-Shot Detection

Pengkai Zhu, Hanxiao Wang, Tolga Bolukbasi, Venkatesh SaligramaIEEE TCSVT 2018 · DOI · OpenReview

Abstract

As we move towards large-scale object detection, it is unrealistic to expect annotated training data, in the form of bounding box annotations around objects, for all object classes at sufficient scale, and so methods capable of unseen object detection are required. We propose a novel zero-shot method based on training an end-to-end model that fuses semantic attribute prediction with visual features to propose object bounding boxes for seen and unseen classes. While we utilize semantic features during training, our method is agnostic to semantic information for unseen classes at test-time. Our method retains the efficiency and effectiveness of YOLOv2 for objects seen during training, while improving its performance for novel and unseen objects. The ability of state-of-art detection methods to learn discriminative object features to reject background proposals also limits their performance for unseen objects. We posit that, to detect unseen objects, we must incorporate semantic information into the visual domain so that the learned visual features reflect this information and leads to improved recall rates for unseen objects. We test our method on PASCAL VOC and MS COCO dataset and observed significant improvements on the average precision of unseen classes.

The Challenge of Open-World Detection

Standard deep learning detectors, such as YOLO or Faster R-CNN, are typically "closed-world" systems. They are optimized to recognize a predefined set of classes and aggressively suppress any other regions as "background." This creates a significant barrier for Zero-Shot Detection (ZSD). If a model has never encountered a specific class during training, its pixels are often treated as part of the background, preventing the model from even proposing a bounding box for the object.

While Zero-Shot Learning (ZSL) has successfully classified cropped images of unseen objects, true detection requires localizing them in complex scenes. The authors address this "background suppression" problem by incorporating semantic information into the visual domain, ensuring the model's learned features can identify "objectness" even for novel categories.

ZS-YOLO: Fusing Visual and Semantic Domains

The proposed architecture, ZS-YOLO, builds upon the YOLOv2 framework with several key innovations designed for the zero-shot regime:

  1. Semantic Prediction Branch: Alongside standard localization, the network includes a branch that predicts a semantic attribute vector (e.g., binary attributes or word2vec embeddings) for every candidate box.
  2. Multi-Modal Confidence Fusion: Unlike standard detectors that predict objectness purely from visual features, ZS-YOLO fuses raw visual features, predicted box coordinates, and predicted semantic attributes to determine confidence.
  3. Composite Loss Function: The model is trained end-to-end using a multi-task loss that includes localization regression, semantic alignment (using cosine similarity with class prototypes), and a confidence loss that rewards regions with meaningful semantic consistency.
ZS-YOLO Architecture and Attribute Prediction

At test time, the model remains agnostic to the specific semantic labels of unseen classes. It utilizes its generalized understanding of what constitutes an object semantically to propose boxes for categories it has never named.

Benchmarking and Key Results

The authors evaluated ZS-YOLO on custom "seen/unseen" splits of the PASCAL VOC and MS COCO datasets. For PASCAL VOC, they utilized the aPY dataset for binary attributes, while for COCO, they employed word2vec embeddings refined through a dimensionality reduction technique (w2vR) to better align with visual features.

Precision-Recall and Unseen Recall Comparison

ZS-YOLO consistently outperformed the baseline YOLOv2 across various metrics:

  • Recall Improvement: On the PASCAL VOC 5/15 split, ZS-YOLO achieved a recall of ~48.2% for unseen classes at a 0.9 confidence threshold, a massive jump from the 18.6% achieved by the standard baseline.
  • Average Precision (AP): On the VOC 10/10 split, the method reached 60.1% AP on unseen data, compared to 56.4% for the baseline. Similar gains were observed on COCO, where ZS-YOLO achieved 43.8% AP on the 60/20 split.
  • Generalization: Qualitative results showed the model successfully detecting unannotated objects (like flags or chameleons) that were not part of the formal "unseen" test list, demonstrating a robust ability to generalize foreground detection.

Conclusion and Future Directions

By teaching the model that "objectness" is a combination of visual patterns and semantic consistency, ZS-YOLO successfully bypasses the background suppression trap of traditional detectors. While the model's increased "curiosity" can lead to a trade-off in precision due to more false positives, the significant boost in recall for unseen objects marks a major step toward open-world detection. Future work may involve integrating this approach into two-stage detectors or utilizing more sophisticated zero-shot classifiers to refine the final labels of proposed boxes.

Summary updated

Abstract

As we move towards large-scale object detection, it is unrealistic to expect annotated training data, in the form of bounding box annotations around objects, for all object classes at sufficient scale, and so methods capable of unseen object detection are required. We propose a novel zero-shot method based on training an end-to-end model that fuses semantic attribute prediction with visual features to propose object bounding boxes for seen and unseen classes. While we utilize semantic features during training, our method is agnostic to semantic information for unseen classes at test-time. Our method retains the efficiency and effectiveness of YOLOv2 for objects seen during training, while improving its performance for novel and unseen objects. The ability of state-of-art detection methods to learn discriminative object features to reject background proposals also limits their performance for unseen objects. We posit that, to detect unseen objects, we must incorporate semantic information into the visual domain so that the learned visual features reflect this information and leads to improved recall rates for unseen objects. We test our method on PASCAL VOC and MS COCO dataset and observed significant improvements on the average precision of unseen classes.
BibTeX
@article{zhu2018zero,
  author = {Pengkai Zhu and Hanxiao Wang and Tolga Bolukbasi and Venkatesh Saligrama},
  title = {{Zero-Shot Detection}},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2018},
  doi = {10.1109/tcsvt.2019.2899569},
}
Figure