Meta-Learning to Detect Rare Objects

Yu-Xiong Wang, Deva Ramanan, Martial HebertICCV 2019 · OpenReview

Abstract

Few-shot learning, i.e., learning novel concepts from few examples, is fundamental to practical visual recognition systems. While most of existing work has focused on few-shot classification, we make a step towards few-shot object detection, a more challenging yet under-explored task. We develop a conceptually simple but powerful meta-learning based framework that simultaneously tackles few-shot classification and few-shot localization in a unified, coherent way. This framework leverages meta-level knowledge about "model parameter generation" from base classes with abundant data to facilitate the generation of a detector for novel classes. Our key insight is to disentangle the learning of category-agnostic and category-specific components in a CNN based detection model. In particular, we introduce a weight prediction meta-model that enables predicting the parameters of category-specific components from few examples. We systematically benchmark the performance of modern detectors in the small-sample size regime. Experiments in a variety of realistic scenarios, including within-domain, cross-domain, and long-tailed settings, demonstrate the effectiveness and generality of our approach under different notions of novel classes.

Introduction

Standard deep learning object detectors, such as Faster R-CNN, typically require thousands of annotated examples to achieve high performance. However, real-world applications often involve "rare" objects where data is scarce. While few-shot classification has seen significant progress, few-shot object detection (FSOD) remains a formidable challenge. Detection requires not only identifying the object category but also precisely localizing it within an image while ignoring complex backgrounds. Previous FSOD methods often treated localization as a category-agnostic task or simply integrated few-shot classifiers into standard detectors, failing to optimize the localization component for low-data scenarios.

The MetaDet Framework

MetaDet addresses these challenges by treating object detection as a meta-learning problem—"learning to learn." The framework is built upon three core pillars:

1. Parameter Disentanglement

The authors disentangle the detection model into two distinct components. The backbone and Region Proposal Network (RPN) are treated as category-agnostic, meaning they are shared across all object classes to capture general visual features and potential object regions. In contrast, the final classification and bounding box regression layers are category-specific, as they must adapt to the unique characteristics of each class.

2. Weight Prediction Meta-Model ($T$)

The central innovation is a weight prediction meta-model, typically implemented as a small multi-layer perceptron (MLP). Instead of traditional fine-tuning, which can lead to overfitting in few-shot settings, this meta-model is trained to predict high-quality, "ideal" parameters for a new class. It takes noisy parameters learned from a few-shot sample as input and transforms them into robust detection weights.

Overview of the MetaDet framework, illustrating the disentanglement of category-agnostic and category-specific parameters and the role of the weight prediction meta-model T.

3. Meta-Training Procedure

The training occurs in two stages. First, a base detector is trained on classes with abundant data. Second, the model undergoes "meta-training," where it simulates numerous few-shot detection tasks (episodes). This process teaches the meta-model how to effectively extrapolate robust parameters from limited data, ensuring that both classification and localization (bounding box regression) are optimized simultaneously.

Experimental Evaluation

MetaDet was evaluated across several rigorous benchmarks, including PASCAL VOC, MS-COCO, and ImageNet (for cross-domain testing). It was also tested on the iNaturalist dataset to assess its performance in long-tailed detection scenarios.

Performance in Low-Data Regimes

MetaDet consistently outperformed standard transfer learning baselines and previous state-of-the-art meta-learning methods. On PASCAL VOC (10-shot), MetaDet achieved 49.6 mAP, a significant improvement over standard transfer learning (~39.5 mAP). The performance gap is even more pronounced in extreme low-data settings, such as 1-shot and 2-shot scenarios.

Performance comparison (mAP) on novel classes across different shot counts (1, 2, 5, 10, 30), showing MetaDet's superiority in low-data regimes.

Key Insights

  • Localization is Category-Specific: A major finding is that bounding box regression parameters benefit significantly from meta-learning. Treating localization as a task that can be "predicted" for new classes leads to more accurate bounding boxes than assuming it is entirely category-agnostic.
  • Robustness and Generality: MetaDet showed strong cross-domain robustness when trained on COCO and tested on ImageNet. Furthermore, in long-tailed settings (iNaturalist), it improved detection for rare "tail" classes without compromising performance on common "head" classes.

Conclusion

MetaDet provides a unified and effective framework for few-shot object detection by leveraging meta-knowledge for parameter generation. By disentangling category-agnostic and category-specific components and employing a weight prediction meta-model, it achieves state-of-the-art results across diverse and challenging scenarios. Future work may explore incorporating unlabeled data or adapting the framework to a wider variety of detector architectures beyond Faster R-CNN.

Summary updated

Abstract

Few-shot learning, i.e., learning novel concepts from few examples, is fundamental to practical visual recognition systems. While most of existing work has focused on few-shot classification, we make a step towards few-shot object detection, a more challenging yet under-explored task. We develop a conceptually simple but powerful meta-learning based framework that simultaneously tackles few-shot classification and few-shot localization in a unified, coherent way. This framework leverages meta-level knowledge about "model parameter generation" from base classes with abundant data to facilitate the generation of a detector for novel classes. Our key insight is to disentangle the learning of category-agnostic and category-specific components in a CNN based detection model. In particular, we introduce a weight prediction meta-model that enables predicting the parameters of category-specific components from few examples. We systematically benchmark the performance of modern detectors in the small-sample size regime. Experiments in a variety of realistic scenarios, including within-domain, cross-domain, and long-tailed settings, demonstrate the effectiveness and generality of our approach under different notions of novel classes.
BibTeX
@inproceedings{wang2019meta,
  author = {Yu-Xiong Wang and Deva Ramanan and Martial Hebert},
  title = {{Meta-Learning to Detect Rare Objects}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=j88DbQGHEJ},
}
Figure