WS-iFSD: Weakly Supervised Incremental Few-shot Object Detection Without Forgetting

Xinyu Gong, Li Yin, Juan-Manuel Perez-Rua, Zhangyang Wang, Zhicheng YanCPAL (Proceedings Track) 2024 · OpenReview

Abstract

Traditional object detection algorithms rely on extensive annotations from a pre-defined set of base categories, leaving them ill-equipped to identify objects from novel classes. We address this limitation by introducing a novel framework for Incremental Few-Shot Object Detection (iFSD). Leveraging a meta-learning approach, our hypernetwork is designed to generate class-specific codes, enabling object recognition from both base and novel categories. To enhance the hypernetwork's generalization performance, we propose a Weakly Supervised Class Augmentation technique that significantly amplifies the training data by merely requiring image-level labels for object localization. Additionally, we stabilize detection performance on base categories by freezing the backbone and detection heads during meta-training. Our model demonstrates significant performance gains on two major benchmarks. Specifically, it outperforms the state-of-the-art ONCE approach on the MS COCO dataset by margins of $2.8\%$ and $20.5\%$ in box AP for novel and base categories, respectively. When trained on MS COCO and cross-evaluated on PASCAL VOC, our model achieves a four-fold improvement in box AP compared to ONCE.

Introduction

Standard object detectors are typically static, restricted to recognizing only the specific classes present in their initial training sets. Expanding these models to include new categories usually necessitates retraining on massive, expertly annotated datasets. Incremental Few-Shot Object Detection (iFSD) seeks to overcome this by enabling models to learn novel classes from as few as 1–10 examples while maintaining performance on original "base" classes.

However, iFSD faces two primary challenges: catastrophic forgetting, where fine-tuning on new data degrades performance on base classes, and poor generalization, where meta-learning models overfit to the limited base classes available during training. The WS-iFSD framework addresses these by combining a frozen detector architecture with a massive, weakly supervised data augmentation strategy.

The four-stage pipeline of WS-iFSD: pre-training, pseudo-bbox generation, meta-training, and meta-testing.

Methodology

The WS-iFSD framework operates in four distinct stages: pre-training the detector, generating pseudo-bounding boxes, meta-training the hypernetwork, and finally meta-testing on both base and novel classes.

Weakly Supervised Class Augmentation

To prevent the hypernetwork from overfitting to a small number of base classes, the authors utilize 1,000 classes from ImageNet. Because ImageNet lacks bounding box annotations, the framework employs Grad-CAM to coarsely localize objects and generate "pseudo-bounding boxes." To ensure data quality, a Mis-Classification Filtering (MCF) step uses a pre-trained classifier to verify images; if the classifier's prediction contradicts the ground-truth label, the image is discarded.

Base-iFSD Architecture

The model is built upon the FCOS (Fully Convolutional One-Stage) detector. During meta-training, the backbone and detection heads are frozen. A lightweight hypernetwork head is trained to generate classification weights for novel classes based on support sets. This freezing strategy effectively eliminates catastrophic forgetting, as the original weights for base classes remain untouched.

Detailed architecture of the WS-iFSD framework, highlighting the data flow between the detection dataset and the augmented classification dataset.

Key Insights

  • Diversity Over Precision: A critical finding is that the hypernetwork generalizes better when trained on 1,000 classes with noisy pseudo-boxes than on 60 classes with perfect human annotations. The variety of objects helps the model learn the general concept of object distinctness.
  • The Effectiveness of Freezing: By keeping the feature extractor robust and frozen, the model avoids the need for complex anti-forgetting penalties or replay buffers.
  • Data Saturation: Benefits from class augmentation begin to saturate at approximately 500 images per class, indicating a "sweet spot" for noisy data in meta-learning.

Results

WS-iFSD demonstrates substantial improvements over the previous state-of-the-art (ONCE):

  • Base Class Stability: On MS COCO, the model improved base class Mean Average Precision (mAP) by 20.5%, confirming the success of the freezing strategy.
  • Novel Class Generalization: The model achieved a 2.8% AP lead over previous methods in 10-shot scenarios on MS COCO.
  • Cross-Dataset Robustness: In evaluations transferring from MS COCO to PASCAL VOC, WS-iFSD achieved a four-fold (4x) improvement in mAP over the ONCE baseline.

Conclusion

WS-iFSD proves that weakly supervised data can significantly bridge the gap in few-shot object detection. While Grad-CAM provides a solid foundation for pseudo-labeling, future work may explore more advanced visual explanation models like Grad-CAM++ to further refine localization. Additionally, scaling the augmentation to larger datasets like ImageNet-21K could potentially push the boundaries of generalization even further.

Summary updated

Abstract

Traditional object detection algorithms rely on extensive annotations from a pre-defined set of base categories, leaving them ill-equipped to identify objects from novel classes. We address this limitation by introducing a novel framework for Incremental Few-Shot Object Detection (iFSD). Leveraging a meta-learning approach, our \hypernetwork is designed to generate class-specific codes, enabling object recognition from both base and novel categories. To enhance the \hypernetwork's generalization performance, we propose a Weakly Supervised Class Augmentation technique that significantly amplifies the training data by merely requiring image-level labels for object localization. Additionally, we stabilize detection performance on base categories by freezing the backbone and detection heads during meta-training. Our model demonstrates significant performance gains on two major benchmarks. Specifically, it outperforms the state-of-the-art ONCE approach on the MS COCO dataset by margins of $2.8\%$ and $20.5\%$ in box AP for novel and base categories, respectively. When trained on MS COCO and cross-evaluated on PASCAL VOC, our model achieves a four-fold improvement in box AP compared to ONCE.
BibTeX
@inproceedings{gong2024ws,
  author = {Xinyu Gong and Li Yin and Juan-Manuel Perez-Rua and Zhangyang Wang and Zhicheng Yan},
  title = {{WS-iFSD: Weakly Supervised Incremental Few-shot Object Detection Without Forgetting}},
  booktitle = {CPAL (Proceedings Track)},
  year = {2024},
  url = {https://openreview.net/forum?id=y2ozeixGaU},
}
Figure