Few-Shot Object Detection via Knowledge Transfer

Geonuk Kim, Honggyu Jung, Seong-Whan LeearXiv 2020 · OpenReview

TL;DR

This paper introduces FSOD-KT, a novel approach to few-shot object detection that leverages prior knowledge of category relationships to improve performance on novel object categories with limited training data. By constructing a knowledge graph and using graph convolutional networks, FSOD-KT effectively transfers information from well-represented "base" categories to scarce "novel" categories, leading to significant improvements over existing methods.

What Problem Does This Paper Address?

Object detection, a cornerstone of computer vision, typically demands vast amounts of annotated data. This reliance becomes a major bottleneck when dealing with "few-shot" scenarios, where only a handful of examples are available for new object categories. Standard object detectors tend to overfit and generalize poorly in such situations. Humans, however, excel at learning new concepts from minimal examples by drawing upon their existing knowledge. This paper aims to bridge this gap by developing a few-shot object detection system that mimics this human ability to transfer and leverage prior knowledge.

Methods

The core of the proposed method, FSOD-KT (Few-Shot Object Detection via Knowledge Transfer), is a prototypical knowledge transfer module integrated with a Faster R-CNN backbone.

Here's a breakdown of the key components:

  • Meta-Learner: This component takes a small "support set" of images containing examples of novel categories (and potentially base categories) and predicts prototypes. These prototypes are vector representations that capture the essence of each category.
  • Knowledge Graph Construction: To guide the prototype prediction, the authors construct a meta-graph. Nodes in this graph represent categories, and edges represent semantic correlations between them. This prior knowledge is derived from the cosine similarity of word embeddings of category names (e.g., "bicycle" is semantically similar to "motorbike").
  • Graph Convolutional Network (GCN): The meta-graph is processed using a GCN. This allows information to propagate between correlated categories, enabling novel category prototypes to benefit from the rich information available for base categories.
  • Prototypical Knowledge Transfer Module: This module uses the GCN output to refine the predicted prototypes. It incorporates a residual connection and a skip connection from the initial prototypes to the meta-classifier. This is crucial to prevent the loss of important information during the aggregation process and to allow prior knowledge to "elastically" guide the visual reasoning.
  • Remodeling R-CNN Predictor Heads: The predicted prototypes are then used to reweight the Region-of-Interest (RoI) features extracted from a "query" image. This reweighting helps the R-CNN predictor heads to focus on features relevant to the novel categories, effectively adapting the detector to new classes.
  • Loss Function: A meta loss is introduced to encourage the predicted prototypes to align with the ground truth labels of the support set. The overall loss combines classification, bounding-box regression, and this meta loss.

Datasets

The experiments were conducted on the PASCAL VOC 2007 dataset. The authors used three different novel/base category split settings to evaluate their method.

Key Insights and Ideas

  • Human-like Learning: The paper draws inspiration from human cognition, where learning new concepts is facilitated by existing knowledge. This is a powerful paradigm shift from purely data-driven approaches.
  • The Power of Category Correlations: Explicitly modeling and leveraging semantic relationships between categories (even those not directly seen in the few-shot examples) is highly effective for improving detection performance on novel classes.
  • Graph-based Knowledge Propagation: Using GCNs on a meta-graph is a robust way to transfer this relational knowledge, allowing for a more nuanced understanding of novel categories.
  • Bridging Prior Knowledge and Data: The authors acknowledge that raw prior knowledge can be noisy or misaligned with the specific training data. Their proposed skip connection mechanism elegantly addresses this by allowing the model to learn how to best integrate this prior knowledge.
  • Maintaining Base Category Performance: A significant challenge in few-shot learning is the potential degradation of performance on previously learned base categories. FSOD-KT demonstrates a marginal drop in base category performance, indicating a more balanced learning approach.

Key Results

  • FSOD-KT significantly outperforms existing few-shot object detection methods (FSRW, Meta R-CNN, FRCN variants) on the PASCAL VOC dataset across various K-shot settings (1, 2, 3, 5, 10 shots).
  • For instance, in the 3-shot setting, FSOD-KT achieves a notable improvement in novel category AP compared to baselines.
  • The method shows a marginal drop in base category performance, which is a crucial advantage over some prior methods that experience significant performance degradation on base classes.
  • Ablation studies confirm the importance of the skip connection, showing a degradation in novel AP when it's removed.
  • Using a randomly generated knowledge graph (instead of semantically derived one) also leads to a performance drop, highlighting the value of meaningful prior knowledge.

Limitations and Future Work

  • Discrepancy between Prior Knowledge and Data: While the skip connection helps, the paper notes that there can still be a discrepancy between textual prior knowledge and the given training data distribution. This suggests that more sophisticated ways to align these might be beneficial.
  • Computational Cost: While not explicitly detailed as a limitation, the use of GCNs and meta-learning components can increase computational complexity compared to simpler detectors.
  • Generalizability to Other Datasets: While PASCAL VOC is a standard benchmark, future work could explore the method's effectiveness on more diverse and challenging datasets.
  • Exploring Richer Prior Knowledge: The current prior knowledge is based on word embeddings. Future work could explore incorporating richer knowledge sources, such as structured knowledge graphs (e.g., WordNet) or attribute-based information.

Summary updated

Abstract

Conventional methods for object detection usually require substantial amounts of training data and annotated bounding boxes. If there are only a few training data and annotations, the object detectors easily overfit and fail to generalize. It exposes the practical weakness of the object detectors. On the other hand, human can easily master new reasoning rules with only a few demonstrations using previously learned knowledge. In this paper, we introduce a few-shot object detection via knowledge transfer, which aims to detect objects from a few training examples. Central to our method is prototypical knowledge transfer with an attached meta-learner. The meta-learner takes support set images that include the few examples of the novel categories and base categories, and predicts prototypes that represent each category as a vector. Then, the prototypes reweight each RoI (Region-of-Interest) feature vector from a query image to remodels R-CNN predictor heads. To facilitate the remodeling process, we predict the prototypes under a graph structure, which propagates information of the correlated base categories to the novel categories with explicit guidance of prior knowledge that represents correlations among categories. Extensive experiments on the PASCAL VOC dataset verifies the effectiveness of the proposed method.
BibTeX
@misc{kim2020few,
  author = {Geonuk Kim and Honggyu Jung and Seong-Whan Lee},
  title = {{Few-Shot Object Detection via Knowledge Transfer}},
  year = {2020},
  eprint = {2008.12496},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2008.12496},
}
Figure